Norm Penalty Research Articles

Simultaneous operating temperature and output power prediction method for photovoltaic modules

Accurate cell temperature and output power prediction are vital for the optimal design and operation of photovoltaic (PV) systems. However, capturing the accurate relationships between cell temperature/circuit parameters and weather conditions is still a challenging task. In this study, a universal radial basis function neural network based hybrid modeling approach is proposed to model the cell temperature and circuit parameters. A simultaneous optimization model with l1 norm penalty is established and a separate parameter estimation strategy is proposed to handle the high computational parameter estimation procedure. The effectiveness of the proposed hybrid modeling approach is validated based on four practical experimental datasets of both commercial and laboratory PV plants. It is thus indicated that the proposed modeling approach could provide a promising potential solution framework for the accurate output power prediction under different PV types and relatively wide weather conditions.

Low-Rank Regression Models for Multiple Binary Responses and their Applications to Cancer Cell-Line Encyclopedia Data

In this article, we study high-dimensional multivariate logistic regression models in which a common set of covariates is used to predict multiple binary outcomes simultaneously. Our work is primarily motivated from many biomedical studies with correlated multiple responses such as the cancer cell-line encyclopedia project. We assume that the underlying regression coefficient matrix is simultaneously low-rank and row-wise sparse. We propose an intuitively appealing selection and estimation framework based on marginal model likelihood, and we develop an efficient computational algorithm for inference. We establish a novel high-dimensional theory for this nonlinear multivariate regression. Our theory is general, allowing for potential correlations between the binary responses. We propose a new type of nuclear norm penalty using the smooth clipped absolute deviation, filling the gap in the related non-convex penalization literature. We theoretically demonstrate that the proposed approach improves estimation accuracy by considering multiple responses jointly through the proposed estimator when the underlying coefficient matrix is low-rank and row-wise sparse. In particular, we establish the non-asymptotic error bounds, and both rank and row support consistency of the proposed method. Moreover, we develop a consistent rule to simultaneously select the rank and row dimension of the coefficient matrix. Furthermore, we extend the proposed methods and theory to a joint Ising model, which accounts for the dependence relationships. In our analysis of both simulated data and the cancer cell line encyclopedia data, the proposed methods outperform the existing methods in better predicting responses. Supplementary materials for this article are available online.

Corrected rank residual constraint model for image denoising

In this paper, a novel image denoising model is proposed, named Corrected Rank Residual Constraint (CRRC). To overcome the drawback that L1 norm penalty yields biased estimators and cannot achieve the best estimation performance, the proposed CRRC incorporates the adaptive correction term with L1 minimization to improve the sparsity of the L1 norm data fidelity term enforced on the rank residual. The existence and uniqueness of solutions of the proposed model are also studied, and the optimization method for solving the model is given. The studies show that the adaptive correction term can not only improve the sparsity of the rank residual but also overcome the over-shrinkage of large singular values. Experimental results demonstrate that the proposed CRRC model outperforms many state-of-the-art methods in both objective and perceptual quality.

Neural network input feature selection using structured l2 − norm penalization

Artificial neural networks are referred to as universal approximators due to their inherent ability to reconstruct complex linear and nonlinear output maps conceived as input-output relationships from data sets. This can be done by reducing large networks via regularization in order to establish compact models containing fewer parameters aimed at describing vital dependencies in data sets. In situations where the data sets contain non-informative input features, devising a continuous, optimal input feature selection technique can lead to improved prediction or classification. We propose a continuous input selection technique through a dimensional reduction mechanism using a ‘structured’ l2 − norm regularization. The implementation is done by identifying the most informative feature subsets from a given data set via an adaptive training mechanism. The adaptation involves introducing a novel, modified gradient approach during training to deal with the non-differentiability associated with the gradient of the structured norm penalty. When the method is applied to process data sets, results indicate that the most informative inputs of artificial neural networks can be selected using a structured l2 − norm penalization.

On Reverse Shrinkage Effects and Shrinkage Overshoot

Given a squared Euclidean norm penalty, we examine some less well-known properties of shrinkage estimates. In particular, we highlight that it is possible for some components of the shrinkage estimator to be placed further away from the prior mean than the original estimate. An analysis of this effect is provided within three different modeling settings—encompassing linear, logistic, and ordinal regression models. Additional simulations show that the outlined effect is not a mathematical artefact, but likely to occur in practice. As a byproduct, they also highlight the possibilities of sign reversals (“overshoots”) for shrinkage estimates. We point out practical consequences and challenges, which might arise from the observed effects with special emphasis on psychometrics.

Hierarchical clustering and matrix completion for the reconstruction of world input–output tables

Multi-regional input–output (I/O) matrices provide the networks of within- and cross-country economic relations. In the context of I/O analysis, the methodology adopted by national statistical offices in data collection raises the issue of obtaining reliable data in a timely fashion and it makes the reconstruction of (parts of) the I/O matrices of particular interest. In this work, we propose a method combining hierarchical clustering and matrix completion with a LASSO-like nuclear norm penalty, to predict missing entries of a partially unknown I/O matrix. Through analyses based on both real-world and synthetic I/O matrices, we study the effectiveness of the proposed method to predict missing values from both previous years data and current data related to countries similar to the one for which current data are obscured. To show the usefulness of our method, an application based on World Input–Output Database (WIOD) tables—which are an example of industry-by-industry I/O tables—is provided. Strong similarities in structure between WIOD and other I/O tables are also found, which make the proposed approach easily generalizable to them.

Mixed degree regression function estimation using weighted lasso penalty

We consider a nonparametric function estimation method with mixed degree using weighted lasso penalty. The ℓ₁ norm penalty controls linear and cubic trends depending on the value of the parameter. In the proposed estimator, we introduce one computational algorithm for constrained convex optimization problems corresponding to the Lagrangian dual problem based on quadratic programming. Subsequently, using simulations and two real data analyses, numerical studies are conducted to verify the performance of the estimator of the proposed method by identifying the relationship between the data.

Regression-based heterogeneity analysis to identify overlapping subgroup structure in high-dimensional data.

Heterogeneity is a hallmark of complex diseases. Regression-based heterogeneity analysis, which is directly concerned with outcome-feature relationships, has led to a deeper understanding of disease biology. Such an analysis identifies the underlying subgroup structure and estimates the subgroup-specific regression coefficients. However, most of the existing regression-based heterogeneity analyses can only address disjoint subgroups; that is, each sample is assigned to only one subgroup. In reality, some samples have multiple labels, for example, many genes have several biological functions, and some cells of pure cell types transition into other types over time, which suggest that their outcome-feature relationships (regression coefficients) can be a mixture of relationships in more than one subgroups, and as a result, the disjoint subgrouping results can be unsatisfactory. To this end, we develop a novel approach to regression-based heterogeneity analysis, which takes into account possible overlaps between subgroups and high data dimensions. A subgroup membership vector is introduced for each sample, which is combined with a loss function. Considering the lack of information arising from small sample sizes, an norm penalty is developed for each membership vector to encourage similarity in its elements. A sparse penalization is also applied for regularized estimation and feature selection. Extensive simulations demonstrate its superiority over direct competitors. The analysis of Cancer Cell Line Encyclopedia data and lung cancer data from The Cancer Genome Atlas show that the proposed approach can identify an overlapping subgroup structure with favorable performance in prediction andstability.

Elastic trend filtering

Abstract Trend filtering aims to estimate underlying trends in time series data, which is necessary to investigate data in a variety of disciplines. We propose a new method called elastic trend filtering. The proposed method combines ℓ 2 and ℓ 1 norm penalties to exploit the benefits and strengths of Hodrick–Prescott and ℓ 1 trend filterings. We apply the alternating direction method of multipliers for its efficient computation and numerical experiments show the soundness and efficiency of the proposed method. We further apply the proposed method to graph cases for potential applications and suggest a trend filtering for its variance estimate.

BIDIMENSIONAL LINKED MATRIX FACTORIZATION FOR PAN-OMICS PAN-CANCER ANALYSIS.

Several modern applications require the integration of multiple large data matrices that have shared rows and/or columns. For example, cancer studies that integrate multiple omics platforms across multiple types of cancer, pan-omics pan-cancer analysis, have extended our knowledge of molecular heterogeneity beyond what was observed in single tumor and single platform studies. However, these studies have been limited by available statistical methodology. We propose a flexible approach to the simultaneous factorization and decomposition of variation across such bidimensionally linked matrices, BIDIFAC+. BIDIFAC+ decomposes variation into a series of low-rank components that may be shared across any number of row sets (e.g., omics platforms) or column sets (e.g., cancer types). This builds on a growing literature for the factorization and decomposition of linked matrices which has primarily focused on multiple matrices that are linked in one dimension (rows or columns) only. Our objective function extends nuclear norm penalization, is motivated by random matrix theory, gives a unique decomposition under relatively mild conditions, and can be shown to give the mode of a Bayesian posterior distribution. We apply BIDIFAC+ to pan-omics pan-cancer data from TCGA, identifying shared and specific modes of variability across four different omics platforms and 29 different cancer types.

A proximal distance algorithm for likelihood-based sparse covariance estimation

Summary This paper addresses the task of estimating a covariance matrix under a patternless sparsity assumption. In contrast to existing approaches based on thresholding or shrinkage penalties, we propose a likelihood-based method that regularizes the distance from the covariance estimate to a symmetric sparsity set. This formulation avoids unwanted shrinkage induced by more common norm penalties, and enables optimization of the resulting nonconvex objective by solving a sequence of smooth, unconstrained subproblems. These subproblems are generated and solved via the proximal distance version of the majorization-minimization principle. The resulting algorithm executes rapidly, gracefully handles settings where the number of parameters exceeds the number of cases, yields a positive-definite solution, and enjoys desirable convergence properties. Empirically, we demonstrate that our approach outperforms competing methods across several metrics, for a suite of simulated experiments. Its merits are illustrated on international migration data and a case study on flow cytometry. Our findings suggest that the marginal and conditional dependency networks for the cell signalling data are more similar than previously concluded.

Stochastic alternating direction method of multipliers for Byzantine-robust distributed learning

This paper aims to solve a distributed learning problem under Byzantine attacks. In the underlying distributed master-worker architecture, there exist a number of unknown but malicious workers that can send arbitrary messages to the master to deviate the learning process, called Byzantine workers. In the literature, a total variation (TV) norm-penalized approximation formulation has been investigated to alleviate the effect of Byzantine attacks. To be specific, the TV norm penalty not only forces the local variables at the regular workers to be close, but is robust to the outliers sent by the Byzantine workers as well. For handling the separable TV norm-penalized approximation formulation, we propose a Byzantine-robust stochastic alternating direction method of multipliers (ADMM). Theoretically, we prove that the proposed method converges to a bounded neighborhood of the optimal solution at a rate of O(1/k) under mild assumptions, where k is the number of iterations and the size of neighborhood is determined by the number of Byzantine workers. Numerical experiments on the MNIST and COVERTYPE datasets further demonstrate the effectiveness of the proposed method to various Byzantine attacks.

Low-rank matrix denoising for count data using unbiased Kullback-Leibler risk estimation

Many statistical studies are concerned with the analysis of observations organized in a matrix form whose elements are count data. When these observations are assumed to follow a Poisson or a multinomial distribution, it is of interest to focus on the estimation of either the intensity matrix (Poisson case) or the compositional matrix (multinomial case) when it is assumed to have a low rank structure. In this setting, it is proposed to construct an estimator minimizing the regularized negative log-likelihood by a nuclear norm penalty. Such an approach easily yields a low-rank matrix-valued estimator with positive entries which belongs to the set of row-stochastic matrices in the multinomial case. Then, as a main contribution, a data-driven procedure is constructed to select the regularization parameter in the construction of such estimators by minimizing (approximately) unbiased estimates of the Kullback-Leibler (KL) risk in such models, which generalize Stein's unbiased risk estimation originally proposed for Gaussian data. The evaluation of these quantities is a delicate problem, and novel methods are introduced to obtain accurate numerical approximation of such unbiased estimates. Simulated data are used to validate this way of selecting regularizing parameters for low-rank matrix estimation from count data. For data following a multinomial distribution, the performances of this approach are also compared to K-fold cross-validation. Examples from a survey study and metagenomics also illustrate the benefits of this methodology for real data analysis.

Underwater Image Enhancement With Hyper-Laplacian Reflectance Priors.

Underwater image enhancement aims at improving the visibility and eliminating color distortions of underwater images degraded by light absorption and scattering in water. Recently, retinex variational models show remarkable capacity of enhancing images by estimating reflectance and illumination in a retinex decomposition course. However, ambiguous details and unnatural color still challenge the performance of retinex variational models on underwater image enhancement. To overcome these limitations, we propose a hyper-laplacian reflectance priors inspired retinex variational model to enhance underwater images. Specifically, the hyper-laplacian reflectance priors are established with the l1/2 -norm penalty on first-order and second-order gradients of the reflectance. Such priors exploit sparsity-promoting and complete-comprehensive reflectance that is used to enhance both salient structures and fine-scale details and recover the naturalness of authentic colors. Besides, the l2 norm is found to be suitable for accurately estimating the illumination. As a result, we turn a complex underwater image enhancement issue into simple subproblems that separately and simultaneously estimate the reflection and the illumination that are harnessed to enhance underwater images in a retinex variational model. We mathematically analyze and solve the optimal solution of each subproblem. In the optimization course, we develop an alternating minimization algorithm that is efficient on element-wise operations and independent of additional prior knowledge of underwater conditions. Extensive experiments demonstrate the superiority of the proposed method in both subjective results and objective assessments over existing methods. The code is available at: https://github.com/zhuangpeixian/HLRP.

Hyperspectral Image Denoising Using Spectral-Spatial Transform-Based Sparse and Low-Rank Representations

This article proposes a denoising method based on sparse spectral–spatial and low-rank representations (SSSLRR) using the 3-D orthogonal transform (3-DOT). SSSLRR can be effectively used to remove the Gaussian and mixed noise. SSSLRR uses 3-DOT to decompose noisy HSI to sparse transform coefficients. The 3-D discrete orthogonal wavelet transform (3-D DWT) is a representative 3-DOT suitable for denoising since it concentrates on the signal in few transform coefficients, and the 3-D discrete orthogonal cosine transform (3-D DCT) is another example. An SSSLRR using 3-D DWT will be called SSSLRR-DWT. SSSLRR-DWT is an iterative algorithm based on the alternating direction method of multipliers (ADMM) that uses sparse and nuclear norm penalties. We use an ablation study to show the effectiveness of the penalties we employ in the method. Both simulated and real hyperspectral datasets demonstrate that SSSLRR outperforms other comparative methods in quantitative and visual assessments to remove the Gaussian and mixed noise.

Unsupervised deep learning with higher-order total-variation regularization for multidimensional seismic data reconstruction

In 3D marine seismic acquisition, the seismic wavefield is not sampled uniformly in the spatial directions. This leads to a seismic wavefield consisting of irregularly and sparsely populated traces with large gaps between consecutive sail lines, especially in the near offsets. The problem of reconstructing the complete seismic wavefield from a subsampled and incomplete wavefield is formulated as an underdetermined inverse problem. We have investigated unsupervised deep learning based on a convolutional neural network for multidimensional wavefield reconstruction of irregularly populated traces defined on a regular grid. Our network is based on an encoder-decoder architecture with an overcomplete latent representation, including appropriate regularization penalties to stabilize the solution. We proposed a combination of penalties, which consists of the [Formula: see text]-norm penalty on the network parameters, and a first- and second-order total-variation penalty on the model. We determined the performance of our method on broadband synthetic data and field data represented by constant-offset gathers from a source-over-cable data set from the Barents Sea. In the field data example, we compare the results to a full production flow from a contractor company, which is based on a 5D Fourier interpolation approach. In this example, our approach displays improved reconstruction of the wavefield with less noise in the sparse near offsets compared with the industry approach, which leads to improved structural definition of the near offsets in the migrated sections.

High-dimensional inference robust to outliers with -norm penalization

This article studies inference in the high-dimensional linear regression model with outliers. Sparsity constraints are imposed on the vector of coefficients of the covariates. The number of outliers can grow with the sample size while their proportion goes to 0. We propose a two-step procedure for inference on the coefficients of a fixed subset of regressors. The first step is a based on several square-root lasso -norm penalized estimators, while the second step is the ordinary least squares estimator applied to a well-chosen regression. We establish asymptotic normality of the two-step estimator. The proposed procedure is efficient in the sense that it attains the semiparametric efficiency bound when applied to the model without outliers under homoscedasticity. This approach is also computationally advantageous, it amounts to solving a finite number of convex optimization programs.

Multivariate log-contrast regression with sub-compositional predictors: Testing the association between preterm infants' gut microbiome and neurobehavioral outcomes.

To link a clinical outcome with compositional predictors in microbiome analysis, the linear log-contrast model is a popular choice, and the inference procedure for assessing the significance of each covariate is also available. However, with the existence of multiple potentially interrelated outcomes and the information of the taxonomic hierarchy of bacteria, a multivariate analysis method that considers the group structure of compositional covariates and an accompanying group inference method are still lacking. Motivated by a study for identifying the microbes in the gut microbiome of preterm infants that impact their later neurobehavioral outcomes, we formulate a constrained integrative multi-view regression. The neurobehavioral scores form multivariate responses, the log-transformed sub-compositional microbiome data form multi-view feature matrices, and a set of linear constraints on their corresponding sub-coefficient matrices ensures the sub-compositional nature. We assume all the sub-coefficient matrices are possible of low-rank to enable joint selection and inference of sub-compositions/views. We propose a scaled composite nuclear norm penalization approach for model estimation and develop a hypothesis testing procedure through de-biasing to assess the significance of different views. Simulation studies confirm the effectiveness of the proposed procedure. We apply the method to the preterm infant study, and the identified microbes are mostly consistent with existing studies and biological understandings.

Dual feature correlation guided multi-task learning for Alzheimer's disease prediction

Alzheimer's disease (AD) is a gradually progressive neurodegenerative disease affecting cognition functions. Predicting the cognitive scores from neuroimage measures and identifying relevant imaging biomarkers are important research topics in the study of AD. Despite machine learning algorithms having many successful applications, the prediction model suffers from the so-called curse of dimensionality. Multi-task feature learning (MTFL) has helped tackle this problem incorporating the correlations among multiple clinical cognitive scores. However, MTFL neglects the inherent correlation among brain imaging measures. In order to better predict the cognitive scores and identify stable biomarkers, we first propose a generalized multi-task formulation framework that incorporates the task and feature correlation structures simultaneously. Second, we present a novel feature-aware sparsity-inducing norm (FAS-norm) penalty to incorporate a useful correlation between brain regions by exploiting correlations among features. Three multi-task learning models that incorporate the FAS-norm penalty are proposed following our framework. Finally, the algorithm based on the alternating direction method of multipliers (ADMM) is developed to optimize the non-smooth problems. We comprehensively evaluate the proposed models on the cross-sectional and longitudinal Alzheimer's disease neuroimaging initiative datasets. The inputs are the thickness measures and the volume measures of the cortical regions of interest. Compared with MTFL, our methods achieve an average decrease of 4.28% in overall error in the cross-sectional analysis and an average decrease of 7.97% in the Alzheimer's Disease Assessment Scale cognitive total score longitudinal analysis. Moreover, our methods identify sensitive and stable biomarkers to physicians, such as the hippocampus, lateral ventricle, and corpus callosum.

Forecasting Trend of Coronavirus Disease 2019 using Multi-Task Weighted TSK Fuzzy System

Artificial intelligence– (AI) based fog/edge computing has become a promising paradigm for infectious disease. Various AI algorithms are embedded in cooperative fog/edge devices to construct medical Internet of Things environments, infectious disease forecast systems, smart health, and so on. However, these systems are usually done in isolation, which is called single-task learning. They do not consider the correlation and relationship between multiple/different tasks, so some common information in the model parameters or data characteristics is lost. In this study, each data center in fog/edge computing is considered as a task in the multi-task learning framework. In such a learning framework, a multi-task weighted Takagi-Sugeno-Kang (TSK) fuzzy system, called MW-TSKFS, is developed to forecast the trend of Coronavirus disease 2019 (COVID-19). MW-TSKFS provides a multi-task learning strategy for both antecedent and consequent parameters of fuzzy rules. First, a multi-task weighted fuzzy c-means clustering algorithm is developed for antecedent parameter learning, which extracts the public information among all tasks and the private information of each task. By sharing the public cluster centroid and public membership matrix, the differences of commonality and individuality can be further exploited. For consequent parameter learning of MW-TSKFS, a multi-task collaborative learning mechanism is developed based on ε-insensitive criterion and L2 norm penalty term, which can enhance the generalization and forecasting ability of the proposed fuzzy system. The experimental results on the real COVID-19 time series show that the forecasting tend model based on multi-task the weighted TSK fuzzy system has a high application value.