Norm Penalty Research Articles

High-Dimensional Estimation of Quadratic Variation Based on Penalized Realized Variance

In this paper, we develop a penalized realized variance (PRV) estimator of the quadratic variation (QV) of a high-dimensional continuous Ito semimartingale. We adapt the principle idea of regularization from linear regression to covariance estimation in a continuous-time high-frequency setting. We show that under a nuclear norm penalization, the PRV is computed by soft-thresholding the eigenvalues of realized variance (RV). It therefore encourages sparsity of singular values or, equivalently, low rank of the solution. We prove our estimator is minimax optimal up to a logarithmic factor. We derive a concentration inequality, which reveals that the rank of PRV is-with a high probability-the number of non-negligible eigenvalues of the QV. Moreover, we also provide the associated non-asymptotic analysis for the spot variance. We suggest an intuitive data-driven bootstrap procedure to select the shrinkage parameter. Our theory is supplemented by a simulation study and an empirical application. The PRV detects about three-five factors in the equity market, with a notable rank decrease during times of distress in financial markets. This is consistent with most standard asset pricing models, where a limited amount of systematic factors driving the cross-section of stock returns are perturbed by idiosyncratic errors, rendering the QV-and also RV-of full rank.

Regularization Strategies for Contiguous and Noncontiguous Damage Detection of Structures

Regularization strategies have attracted attention in the structural damage detection (SDD) field. However, there is lack of studies on regularization strategies for damage patterns in the existing methods. This paper proposes regularization strategies for contiguous and noncontiguous damages of structures and performs comparative studies. The objective functions are first defined to consider effects of strategies on SDD by adding distinct norm penalties, and then are solved by the particle swarm optimization (PSO). Three numerical simulation models are employed to assess the applicability of three strategies. The results show that the [Formula: see text] norm regularization is suitable for detecting multiple damages, the [Formula: see text] norm regularization performs well in contiguous damages, and the sparsest solutions can be obtained by the [Formula: see text] norm regularization.

An Efficient Algorithm to Compute Norms for Finite Horizon, Linear Time-Varying Systems

We present an efficient algorithm to compute the induced norms of finite-horizon Linear Time-Varying (LTV) systems. The formulation includes both induced L2 and terminal Euclidean norm penalties on outputs. Existing computational approaches include the power iteration and the bisection of a Riccati Differential Equation (RDE). The power iteration has low computation time per iteration but overall convergence can be slow. In contrast, the RDE condition provides guaranteed bounds on the induced gain but single RDE integration can be slow. The complementary features of these two algorithms are combined to develop a new algorithm that is both fast and provides provable upper and lower bounds on the induced norm within the desired tolerance. The algorithm also provides a worst-case disturbance input that achieves the lower bound on the norm. We also present a new proof which shows that the power iteration for this problem converges monotonically. Finally, we show a controllability Gramian based simpler computational method for induced L <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> -to-Euclidean norm. This can be used to compute the reachable set at any time on the horizon. Numerical examples are provided to demonstrate the proposed algorithm.

Generalized simultaneous component analysis of binary and quantitative data

AbstractIn the current era of systems biology research, there is a need for the integrative analysis of binary and quantitative genomics data sets measured on the same objects. One standard tool of exploring the underlying dependence structure present in multiple quantitative data sets is the simultaneous component analysis (SCA) model. However, it does not have any provisions when a part of the data are binary. To this end, we propose the generalized SCA (GSCA) model, which takes into account the distinct mathematical properties of binary and quantitative measurements in the maximum likelihood framework. Like in the SCA model, a common low‐dimensional subspace is assumed to represent the shared information between these two distinct types of measurements. To achieve a low rank solution, we propose to use a concave variant of the nuclear norm penalty. An efficient majorization algorithm is developed to fit this model with different concave penalties. Realistic simulations (low signal‐to‐noise ratio and highly imbalanced binary data) are used to evaluate the performance of the proposed model in recovering the underlying structure. Also, a missing value based cross‐validation procedure is implemented for model selection. We illustrate the usefulness of the GSCA model for exploratory data analysis of quantitative gene expression and binary copy number aberration measurements obtained from the GDSC1000 data sets.

Multi-View Multi-Label Learning With Sparse Feature Selection for Image Annotation

In image analysis, image samples are always represented by multiple view features and associated with multiple class labels for better interpretation. However, multiple view data may include noisy, irrelevant and redundant features, while multiple class labels can be noisy and incomplete. Due to the special data characteristic, it is hard to perform feature selection on multi-view multi-label data. To address these challenges, in this paper, we propose a novel multi-view multi-label sparse feature selection (MSFS) method, which exploits both view relations and label correlations to select discriminative features for further learning. Specifically, the multi-labeled information is decomposed into a reduced latent label representation to capture higher level concepts and correlations among multiple labels. Multiple local geometric structures are constructed to exploit visual similarities and relations for different views. By taking full advantage of the latent label representation and multiple local geometric structures, the sparse regression model with an $l_{2,1}$ -norm and an Frobenius norm ( $F$ -norm) penalty terms is utilized to perform hierarchical feature selection, where the $F$ -norm penalty performs high-level (i.e., view-wise) feature selection to preserve the informative views and the $l_{2,1}$ -norm penalty conducts low-level (i.e., row-wise) feature selection to remove noisy features. To solve the proposed formulation, we also devise a simple yet efficient iterative algorithm. Experiments and comparisons on real-world image datasets demonstrate the effectiveness and potential of MSFS.

An Accurate Sparse SAR Imaging Method for Enhancing Region-Based Features Via Nonconvex and TV Regularization

With the rapid development of compressed sensing theories and applications, sparse signal processing has been widely used in synthetic aperture radar (SAR) imaging during the recent years. As an efficient tool for sparse reconstruction, ℓ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> optimization induces sparsity the most effectively, and the ℓ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> -norm penalty is usually combined with the total variation norm (TV-norm) penalty to construct a compound regularizer in order to enhance the point-based features as well as the region-based features. However, as a convex optimizer, the analytic solution of ℓ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> regularization-based sparse signal reconstruction is usually a biased estimation. Aiming at this issue, in this article, we quantitatively analyzed the variation of reconstruction bias with respect to the complex reflectivity of targets, the undersampling ratio and the noise power. In order to reduce the bias effect and improve the reconstruction accuracy, we adopted the nonconvex regularization-based sparse SAR imaging method with a nonconvex penalty family. Furthermore, we linearly combined the nonconvex penalty and the TV-norm penalty to form a compound regularizer in the imaging model, which can improve the reconstruction accuracy of distributed targets and maintain the continuity of the backscattering coefficient. Simulation results showed that compared with ℓ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> regularization, nonconvex regularization can reduce the average relative bias from 10.88% to 0.25%; compared with the matched filtering method and ℓ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> and TV regularization, nonconvex & TV regularization can reduce the variance of the uniformly distributed targets by 80% without losing of reconstruction accuracy. Experiments on Gaofen-3 SAR data are also exploited to verify the effectiveness of the proposed method.

High-dimensional dynamic systems identification with additional constraints

This note presents a unified analysis of the identification of dynamical systems with low-rank constraints under high-dimensional scaling. This identification problem for dynamic systems is challenging due to the intrinsic dependency of the data. To alleviate this problem, we first formulate this identification problem into a multivariate linear regression problem with a row-sub-Gaussian measurement matrix using the more general input designs and the independently repeated sampling schemes. We then propose a nuclear norm heuristic method that estimates the parameter matrix of a dynamic system from a few input-state data samples. Based on this, we can extend the existing results. In this article, we consider two scenarios. (i) In the noiseless scenario, nuclear-norm minimization is introduced for promoting low-rank. We define the notion of weak restricted isometry property, which is weaker than the ordinary restricted isometry property, and show it holds with high probability for the row-sub-Gaussian measurement matrix. Thereby, the rank-minimization matrix can be exactly recovered from a finite number of data samples. (ii) In the noisy scenario, a regularized framework involving the nuclear norm penalty is established. We give the notion of operator norm curvature condition for the loss function and show it holds for the row-sub-Gaussian measurement matrix with high probability. Consequently, when specifying the suitable choice of the regularization parameter, the operator norm error of the optimal solution of this program has a sharp bound given a finite amount of data samples. This operator norm error bound is always smaller than the ordinary Frobenius norm error bound obtained in the existing work.

DOA Estimation With Small Snapshots Using Weighted Mixed Norm Based on Spatial Filter

$\ell _{2,1}$ -norm penalized compressive sensing (CS) is utilized to improve the performance of DOA estimation with small snapshots recently. However, the existing CS-based methods are not robust to the noise. In this article, we propose a CS-based DOA estimation using a novel weighted $\ell _{2,1}$ -norm penalty. A spatial filter which can roughly “clean up” or eliminate the signals coming from the directions of the true sources is constructed. Thus, the space spectrum of the spatial filter can work as a weighting matrix to adjust the sparse penalty automatically. A new weighted $\ell _{2,1}$ -norm penalty based on this spatial filter is then proposed for DOA estimation. Simulation results demonstrate the effectiveness and efficiency of the proposed algorithm.

Application of a class of iterative algorithms and their accelerations to Jacobian-based linearized EIT image reconstruction

ABSTRACT This work is concerned with the image reconstruction of the Jacobian-based linearized EIT problem. Based on the homotopy perturbation technology, we first propose a novel class of iteration schemes with different orders of approximation truncation (named as HPI for short), which contains Landweber-type iteration method. Afterwards, nonsmooth priors such as -norm or total variation penalty are introduced with the proposed HPI method to improve the imaging quality. However, it is known that Landweber-type iteration method is the widely used imaging algorithm, but in its basic form, the imaging speed is slow and the imaging accuracy is low. Furthermore, Nesterov's strategy is conducted to accelerate the proposed approaches. Numerical simulations with synthetic dada are performed to validate that our proposed approaches have robustness to noise and can indeed improve the image resolution and the imaging speed. Especially, numerical results explicitly show that Nesterov's acceleration versions make significantly remarkable acceleration effects.

Non-convex ℓ p regularization for sparse reconstruction of electrical impedance tomography

This work is to investigate the image reconstruction of electrical impedance tomography from the electrical measurements made on an object's surface. An -norm (0<p<1) sparsity-promoting regularization is considered to deal with the fully non-linear electrical impedance tomography problem, and a novel type of smoothing gradient-type iteration scheme is introduced. To avoid the difficulty in calculating its gradient in the optimization process, a smoothing Huber potential function is utilized to approximate the -norm penalty. We then propose the smoothing algorithm in the general frame and establish that any accumulation point of the generated iteration sequence is a first-order stationary point of the original problem. Furthermore, one iteration scheme based on the homotopy perturbation technology is derived to find the minimizers of the Huberized approximated objective function. Numerical experiments show that non-convex -norm sparsity-promoting regularization improves the spatial resolution and is more robust with respect to noise, in comparison with -norm regularization.

Anti-sparse representation for structural model updating using l ∞ norm regularization

Finite element (FE) model based structural damage detection (SDD) methods play vital roles in effectively locating and quantifying structural damages. Among these methods, structural model updating should be conducted before SDD to obtain benchmark models of real structures. However, the characteristics of updating parameters are not reasonably considered in existing studies. Inspired by the lll∞ norm regularization, a novel anti-sparse representation method is proposed for structural model updating in this study. Based on sensitivity analysis, both frequencies and mode shapes are used to define an objective function at first. Then, by adding l∞ norm penalty, an optimization problem is established for structural model updating. As a result, the optimization problem can be solved by the fast iterative shrinkage thresholding algorithm (FISTA). Moreover, comparative studies with classical regularization strategy, i.e. the l2 norm regularization method, are conducted as well. To intuitively illustrate the effectiveness of the proposed method, a 2-DOF spring-mass model is taken as an example in numerical simulations. The updating results show that the proposed method has a good robustness to measurement noises. Finally, to further verify the applicability of the proposed method, a six-storey aluminum alloy frame is designed and fabricated in laboratory. The added mass on each storey is taken as updating parameter. The updating results provide a good agreement with the true values, which indicates that the proposed method can effectively update the model parameters with a high accuracy.

A Non-Convex L₁-Norm Penalty-Based Total Generalized Variation Model for Reconstruction of Conductivity Distribution

Electrical impedance tomography (EIT) is a potential imaging technique for reconstructing the interior conductivity distribution within an object. The reconstruction is realized by processing the boundary voltage measured from the EIT. Mathematically, the reconstruction of conductivity distribution is a highly nonlinear ill-posed inverse problem as the solution is not unique and is sensitive to the noise, which largely hinders the practical application of the EIT technique. In this paper, a novel modified non-convex ${L}_{1}$ -norm penalty based total generalized variation (NCP-TGV) model is proposed to solve the inverse problem and reconstruct the conductivity distribution. An iteratively reweighted ${L}_{1}$ algorithm is developed to convert the non-convex model to convex function and the Chambolle-Pock primal-dual algorithm is then applied to solve the minimization problem. To demonstrate the performance of the proposed NCP-TGV model in reconstructing the conductivity distribution, extensive numerical simulation and experimental work have been carried out. Additionally, the proposed approach has been validated by comparing its performance with the convex ${L}_{1}$ -norm penalty based total variation (TV) and TGV inverse models. The results indicate that the images reconstructed by the proposed NCP-TGV model show better quality with the advantages of simultaneous reduction of staircase effect and preservation of edge in comparison to the TV and TGV approaches.

Logistic principal component analysis via non-convex singular value thresholding

Multivariate binary data is becoming abundant in current biological research. Logistic principal component analysis (PCA) is one of the commonly used tools to explore the relationships inside a multivariate binary data set by exploiting the underlying low rank structure. We re-expressed the logistic PCA model based on the latent variable interpretation of the generalized linear model on binary data. The multivariate binary data set is assumed to be the sign observation of an unobserved quantitative data set, on which a low rank structure is assumed to exist. However, the standard logistic PCA model (using exact low rank constraint) is prone to overfitting, which could lead to the divergence of some estimated parameters towards infinity. We propose to fit a logistic PCA model through non-convex singular value thresholding to alleviate the overfitting issue. An efficient Majorization-Minimization algorithm is implemented to fit the model and a missing value based cross validation (CV) procedure is introduced for the model selection. Our experiments on realistic simulations of imbalanced binary data and low signal-to-noise ratio show that the CV error based model selection procedure is successful in selecting the proposed model. Furthermore, the selected model demonstrates superior performance in recovering the underlying low rank structure compared to models with convex nuclear norm penalty and exact low rank constraint. Finally, a binary copy number aberration data set is used to illustrate the proposed methodology in practice.

Logistic regression with image covariates via the combination of L1 and Sobolev regularizations.

The use of image covariates to build a classification model has lots of impact in various fields, such as computer science, medicine, and so on. The aim of this paper is to develop an estimation method for logistic regression model with image covariates. We propose a novel regularized estimation approach, where the regularization is a combination of L1 regularization and Sobolev norm regularization. The L1 penalty can perform variable selection, while the Sobolev norm penalty can capture the shape edges information of image data. We develop an efficient algorithm for the optimization problem. We also establish a nonasymptotic error bound on parameter estimation. Simulated studies and a real data application demonstrate that our proposed method performs very well.

Estimation of Genomic Breed Composition for Purebred and Crossbred Animals Using Sparsely Regularized Admixture Models.

A variety of statistical methods, such as admixture models, have been used to estimate genomic breed composition (GBC). These methods, however, tend to produce non-zero components to reference breeds that shared some genomic similarity with a test animal. These non-essential GBC components, in turn, offset the estimated GBC for the breed to which it belongs. As a result, not all purebred animals have 100% GBC of their respective breeds, which statistically indicates an elevated false-negative rate in the identification of purebred animals with 100% GBC as the cutoff. Otherwise, a lower cutoff of estimated GBC will have to be used, which is arbitrary, and the results are less interpretable. In the present study, three admixture models with regularization were proposed, which produced sparse solutions through suppressing the noise in the estimated GBC due to genomic similarities. The regularization or penalty forms included the L1 norm penalty, minimax concave penalty (MCP), and smooth clipped absolute deviation (SCAD). The performances of these regularized admixture models on the estimation of GBC were examined in purebred and composite animals, respectively, and compared to that of the non-regularized admixture model as the baseline model. The results showed that, given optimal values for λ, the three sparsely regularized admixture models had higher power and thus reduced the false-negative rate for the breed identification of purebred animals than the non-regularized admixture model. Of the three regularized admixture models, the two with a non-convex penalty outperformed the one with L1 norm penalty. In the Brangus, a composite cattle breed, estimated GBC were roughly comparable among the four admixture models, but all the four models underestimated the GBC for these composite animals when non-ancestral breeds were included as the reference. In conclusion, the admixture models with sparse regularization gave more parsimonious, consistent and interpretable results of estimated GBC for purebred animals than the non-regularized admixture model. Nevertheless, the utility of regularized admixture models for estimating GBC in crossbred or composite animals needs to be taken with caution.

Reweighted l p constraint LMS‐based adaptive sparse channel estimation for cooperative communication system

The issue of sparsity adaptive channel reconstruction in time-varying cooperative communication networks through the amplify-and-forward transmission scheme is studied. A new sparsity adaptive system identification method is proposed, namely reweighted l p norm ( 0 < p < 1 ) penalised least mean square (LMS) algorithm. The main idea of the algorithm is to add a l p norm penalty of sparsity into the cost function of the LMS algorithm. By doing so, the weight factor becomes a balance parameter of the associated l p norm adaptive sparse system identification. Subsequently, the steady state of the coefficient misalignment vector is derived theoretically, with a performance upper bounds provided which serve as a sufficient condition for the LMS channel estimation of the precise reweighted l p norm. With the upper bounds, the authors prove that the l p ( 0 < p < 1 ) norm sparsity inducing cost function is superior to the reweighted l 1 norm. An optimal selection of p for the l p norm problem is studied to recover various d sparse channel vectors. Several experiments verify that the simulation results agree well with the theoretical analysis, and thus demonstrate that the proposed algorithm has a better convergence speed and better steady-state behaviour than other LMS algorithms.

A novel block-sparse proportionate NLMS algorithm based on the l2,0 norm

To ameliorate the performance of the normalized least mean square (NLMS) algorithm applied in identifying block-sparse system, a novel block-sparse proportionate NLMS (BS-PNLMS) algorithm based on the mixed l2,0 norm is presented in this paper. The cost function of proposed algorithm is obtained by introducing the approximated mixed l2,0 norm penalty in the cost function of BS-PNLMS algorithm, which is equivalent to attaching the mixed l2,0 norm zero attraction to the update equation. The proposed algorithm is named as the l2,0-BS-PNLMS algorithm. Futhermore, a variable step-size l2,0-BS-PNLMS is developed to further improve the performance of the l2,0-BS-PNLMS. Simulation results demonstrate the advantages of proposed algorithms.

Reweighted generalized minimax-concave sparse regularization and application in machinery fault diagnosis

The vibration signal of faulty rotating machinery tends to be a mixture of repetitive transients, discrete frequency components and noise. How to accurately extract the repetitive transients is a critical issue for machinery fault diagnosis. Inspired by reweighted L1 (ReL1) minimization for sparsity enhancement, a reweighted generalized minimax-concave (ReGMC) sparse regularization method is proposed to extract the repetitive transients. We utilize the generalized minimax-concave (GMC) penalty to regularize the weighted sparse representation model to overcome the underestimation deficiency of L1 norm penalty. Moreover, a new reweight strategy which is different from the reweight strategy in ReL1 for sparsity enhancement is proposed according to the statistical characteristic, i.e., squared envelope spectrum kurtosis. Then ReGMC is proposed by solving a series of weighted GMC minimization problems. ReGMC is utilized to process a simulated signal and the vibration signals of a hot-milling transmission gearbox and a run-to-failure bearing with incipient fault. The ReGMC analysis results and the comparison studies show that ReGMC can effectively extract the repetitive transients while suppressing the discrete frequency components and noise, and behaves better than GMC, improved lasso, and spectral kurtosis.

A deep learning approach for prediction of Parkinson's disease progression.

This paper proposes a deep neural network (DNN) model using the reduced input feature space of Parkinson's telemonitoring dataset to predict Parkinson's disease (PD) progression. PD is a chronic and progressive nervous system disorder that affects body movement. PD is assessed by using the unified Parkinson's disease rating scale (UPDRS). In this paper, firstly, principal component analysis (PCA) is employed to the featured dataset to address the multicollinearity problems in the dataset and to reduce the dimension of input feature space. Then, the reduced input feature space is fed into the proposed DNN model with a tuned parameter norm penalty (L2) and analyses the prediction performance of it in PD progression by predicting Motor and Total-UPDRS score. The model's performance is evaluated by conducting several experiments and the result is compared with the result of previously developed methods on the same dataset. The model's prediction accuracy is measured by fitness parameters, mean absolute error (MAE), root mean squared error (RMSE), and coefficient of determination (R2). The MAE, RMSE, and R2 values are 0.926, 1.422, and 0.970 respectively for motor-UPDRS. These values are 1.334, 2.221, and 0.956 respectively for Total-UPDRS. Both the Motor and Total-UPDRS score is better predicted by the proposed method. This paper shows the usefulness and efficacy of the proposed method for predicting the UPDRS score in PD progression.

Synchronous Parallel Block Coordinate Descent Method for Nonsmooth Convex Function Minimization

This paper proposes a synchronous parallel block coordinate descent algorithm for minimizing a composite function, which consists of a smooth convex function plus a non-smooth but separable convex function. Due to the generalization of the proposed method, some existing synchronous parallel algorithms can be considered as special cases. To tackle high dimensional problems, the authors further develop a randomized variant, which randomly update some blocks of coordinates at each round of computation. Both proposed parallel algorithms are proven to have sub-linear convergence rate under rather mild assumptions. The numerical experiments on solving the large scale regularized logistic regression with l1 norm penalty show that the implementation is quite efficient. The authors conclude with explanation on the observed experimental results and discussion on the potential improvements.