Norm Regularization Research Articles

Total Third-Degree Variation for Noise Reduction in Atomic-Resolution STEM Images.

Scanning Transmission Electron Microscopy (STEM) enables direct determination of atomic arrangements in materials and devices. However, materials such as battery components are weak for electron beam irradiation and low electron doses are required to prevent beam-induced damages. Noise removal is thus essential for precise structural analysis of electron beam sensitive materials at atomic resolution. Total square variation (TSV) regularization is an algorithm that exhibits high noise removal performance. However, the use of the TSV regularization term leads to significant image blurring and intensity reduction. To address these problems, we here propose a new approach adopting L2 norm regularization based on higher-order total variation. An atomic-resolution STEM image can be approximated as a set of smooth curves represented by quadratic functions. Since the third-degree derivative of any quadratic function is 0, total third-degree variation (TTDV) is suitable for a regularization term. The application of TTDV for denoising the atomic-resolution STEM image of CaF2 observed along the [001] zone axis is shown, where we can clearly see the Ca and F atomic columns without compromising image quality.

Fast Sparse Discriminative K-Means for Unsupervised Feature Selection.

Embedded feature selection approach guides subsequent projection matrix (selection matrix) learning through the acquisition of pseudolabel matrix to conduct feature selection tasks. Yet the continuous pseudolabel matrix learned from relaxed problem based on spectral analysis deviates from reality to some extent. To cope with this issue, we design an efficient feature selection framework inspired by classical least-squares regression (LSR) and discriminative K-means (DisK-means), which is called the fast sparse discriminative K-means (FSDK) for the feature selection method. First, the weighted pseudolabel matrix with discrete trait is introduced to avoid trivial solution from unsupervised LSR. On this condition, any constraint imposed into pseudolabel matrix and selection matrix is dispensable, which is significantly beneficial to simplify the combinational optimization problem. Second, the l2,p -norm regularizer is introduced to satisfy the row sparsity of selection matrix with flexible p . Consequently, the proposed FSDK model can be treated as a novel feature selection framework integrated from the DisK-means algorithm and l2,p -norm regularizer to optimize the sparse regression problem. Moreover, our model is linearly correlated with the number of samples, which is speedy to handle the large-scale data. Comprehensive tests on various data terminally illuminate the effectiveness and efficiency of FSDK.

Low-Rank Tensor Regularized Views Recovery for Incomplete Multiview Clustering.

In real applications, it is often that the collected multiview data contain missing views. Most existing incomplete multiview clustering (IMVC) methods cannot fully utilize the underlying information of missing data or sufficiently explore the consistent and complementary characteristics. In this article, we propose a novel Low-rAnk Tensor regularized viEws Recovery (LATER) method for IMVC, which jointly reconstructs and utilizes the missing views and learns multilevel graphs for comprehensive similarity discovery in a unified model. The missing views are recovered from a common latent representation, and the recovered views conversely improve the learning of shared patterns. Based on the shared subspace representations and recovered complete multiview data, the multilevel graphs are learned by self-representation to fully exploit the consistent and complementary information among views. Besides, a tensor nuclear norm regularizer is introduced to pursue the global low-rank property and explore the interview correlations. An alternating direction minimization algorithm is presented to optimize the proposed model. Moreover, a new initialization method is proposed to promote the effectiveness of our method for latent representation learning and missing data recovery. Extensive experiments demonstrate that our method outperforms the state-of-the-art approaches.

Representations and generalization in artificial and brain neural networks

Humans and animals excel at generalizing from limited data, a capability yet to be fully replicated in artificial intelligence. This perspective investigates generalization in biological and artificial deep neural networks (DNNs), in both in-distribution and out-of-distribution contexts. We introduce two hypotheses: First, the geometric properties of the neural manifolds associated with discrete cognitive entities, such as objects, words, and concepts, are powerful order parameters. They link the neural substrate to the generalization capabilities and provide a unified methodology bridging gaps between neuroscience, machine learning, and cognitive science. We overview recent progress in studying the geometry of neural manifolds, particularly in visual object recognition, and discuss theories connecting manifold dimension and radius to generalization capacity. Second, we suggest that the theory of learning in wide DNNs, especially in the thermodynamic limit, provides mechanistic insights into the learning processes generating desired neural representational geometries and generalization. This includes the role of weight norm regularization, network architecture, and hyper-parameters. We will explore recent advances in this theory and ongoing challenges. We also discuss the dynamics of learning and its relevance to the issue of representational drift in the brain.

A Novel Fast Iterative STAP Method with a Coprime Sampling Structure.

In space-time adaptive processing (STAP), the coprime sampling structure can obtain better clutter suppression capabilities at a lower hardware cost than the uniform linear sampling structure. However, in practical applications, the performance of the algorithm is often limited by the number of training samples. To solve this problem, this paper proposes a fast iterative coprime STAP algorithm based on truncated kernel norm minimization (TKNM). This method establishes a virtual clutter covariance matrix (CCM), introduces truncated kernel norm regularization technology to ensure the low rank of the CCM, and transforms the non-convex problem into a convex optimization problem. Finally, a fast iterative solution method based on the alternating direction method is presented. The effectiveness and accuracy of the proposed algorithm are verified through simulation experiments.

Joint Anchor Graph Embedding and Discrete Feature Scoring for Unsupervised Feature Selection.

The success of existing unsupervised feature selection (UFS) methods heavily relies on the assumption that the intrinsic relationships among original high-dimensional (HD) data samples exist in the discriminative low-dimension (LD) subspace. However, previous UFS methods commonly construct pairwise graphs and employ l2,1 -norm regularization to severally preserve the local structure and calculate the score of features, which is computationally complex and easy to get stuck into local optimum, so that those approaches cannot be applied in dealing with large-scale datasets in practice. To overcome this challenge, we propose a novel UFS method, in which a novel anchor graph embedding paradigm is designed to extract the local neighborhood relationships among data samples by reducing the computational complexity of graph construction to be linear in the number of data. Moreover, to improve the optimality of selected features as well as the performance of downstream tasks, we propose a discrete feature scoring mechanism, which imposes orthogonal l2,0 -norm constraints on learned projections, in order to enhance the distinction of feature scores as well as reduce the probability of falling into local optimum. In addition, solving the proposed nonconvex and nonsmooth NP-hard problem is challenging, and we present an efficient optimization algorithm to address it and acquire a closed-form solution of the transformation matrix. Extensive experiments demonstrate the effectiveness and efficiency of the proposed UFS by comparison with several state-of-the-art approaches to clustering and image segmentation tasks.

3D inversion of gravity data using mixed Lp–Lq norm regularization, a case study for potash targeting at the Ghare-Aghaj deposit in Iran

3D inversion of gravity data using mixed Lp–Lq norm regularization, a case study for potash targeting at the Ghare-Aghaj deposit in Iran

Accurate detection method of traveling wave shape based on EEMD and L1 norm regularization

Aiming at the problem of waveform distortion of the secondary traveling wave obtained from the primary traveling wave signal transmitted by the voltage traveling wave sensor, an accurate detection method of voltage traveling wave based on ensemble empirical mode decomposition (EEMD) and L1 norm least squares regularization is proposed in this paper. Firstly, the forward transfer function model of the specialized voltage traveling wave sensor is established, and the inversion model is derived to analyze the phase frequency and amplitude frequency characteristics of the secondary traveling wave after transmission. EEMD is used to decompose the secondary traveling wave into the intrinsic modal function (IMF) components of different frequency bands. The L1 norm least square regularization inversion algorithm was used to invert each IMF component. Finally, the IMF' components after inversion are synthesized to obtain the primary traveling wave. Simulation and experimental results demonstrate that the proposed method is not affected by noise, mode aliasing effect and regularization parameter accuracy, and the obtained waveform has high similarity to the simulated waveform, which enables precise identification of fault traveling wave.

EEG functional connectivity as a Riemannian mediator: An application to malnutrition and cognition.

Mediation analysis assesses whether an exposure directly produces changes in cognitive behavior or is influenced by intermediate "mediators". Electroencephalographic (EEG) spectral measurements have been previously used as effective mediators representing diverse aspects of brain function. However, it has been necessary to collapse EEG measures onto a single scalar using standard mediation methods. In this article, we overcome this limitation and examine EEG frequency-resolved functional connectivity measures as a mediator using the full EEG cross-spectral tensor (CST). Since CST samples do not exist in Euclidean space but in the Riemannian manifold of positive-definite tensors, we transform the problem, allowing for the use of classic multivariate statistics. Toward this end, we map the data from the original manifold space to the Euclidean tangent space, eliminating redundant information to conform to a "compressed CST." The resulting object is a matrix with rows corresponding to frequencies and columns to cross spectra between channels. We have developed a novel matrix mediation approach that leverages a nuclear norm regularization to determine the matrix-valued regression parameters. Furthermore, we introduced a global test for the overall CST mediation and a test to determine specific channels and frequencies driving the mediation. We validated the method through simulations and applied it to our well-studied 50+-year Barbados Nutrition Study dataset by comparing EEGs collected in school-age children (5-11 years) who were malnourished in the first year of life with those of healthy classmate controls. We hypothesized that the CST mediates the effect of malnutrition on cognitive performance. We can now explicitly pinpoint the frequencies (delta, theta, alpha, and beta bands) and regions (frontal, central, and occipital) in which functional connectivity was altered in previously malnourished children, an improvement to prior studies. Understanding the specific networks impacted by a history of postnatal malnutrition could pave the way for developing more targeted and personalized therapeutic interventions. Our methods offer a versatile framework applicable to mediation studies encompassing matrix and Hermitian 3D tensor mediators alongside scalar exposures and outcomes, facilitating comprehensive analyses across diverse research domains.

Generalized and Robust Least Squares Regression.

As a simple yet effective method, least squares regression (LSR) is extensively applied for data regression and classification. Combined with sparse representation, LSR can be extended to feature selection (FS) as well, in which l1 regularization is often applied in embedded FS algorithms. However, because the loss function is in the form of squared error, LSR and its variants are sensitive to noises, which significantly degrades the effectiveness and performance of classification and FS. To cope with the problem, we propose a generalized and robust LSR (GRLSR) for classification and FS, which is made up of arbitrary concave loss function and the l2,p -norm regularization term. Meanwhile, an iterative algorithm is applied to efficiently deal with the nonconvex minimization problem, in which an additional weight to suppress the effect of noises is added to each data point. The weights can be automatically assigned according to the error of the samples. When the error is large, the value of the corresponding weight is small. It is this mechanism that allows GRLSR to reduce the impact of noises and outliers. According to the different formulations of the concave loss function, four specific methods are proposed to clarify the essence of the framework. Comprehensive experiments on corrupted datasets have proven the advantage of the proposed method.

Spatial Invariant Tensor Self-Representation Model for Hyperspectral Anomaly Detection.

With the development of hyperspectral imaging technology, the hyperspectral anomaly has attracted considerable attention due to its significant role in many applications. Hyperspectral images (HSIs) with two spatial dimensions and one spectral dimension are intrinsically three-order tensors. However, most of the existing anomaly detectors were designed after converting the 3-D HSI data into a matrix, which destroys the multidimension structure. To solve this problem, in this article, we propose a spatial invariant tensor self-representation (SITSR) hyperspectral anomaly detection algorithm, which is derived based on the tensor-tensor product (t-product) to preserve the multidimension structure and achieve a comprehensive description of the global correlation of HSIs. Specifically, we exploit the t-product to integrate spectral information and spatial information, and the background image of each band is represented as the sum of the t-product of all bands and their corresponding coefficients. Considering the directionality of the t-product, we utilize two tensor self-representation methods with different spatial modes to obtain a more balanced and informative model. To depict the global correlation of the background, we merge the unfolding matrices of two representative coefficients and constrain them to lie in a low-dimensional subspace. Moreover, the group sparsity of anomaly is characterized by l2.1.1 norm regularization to promote the separation of background and anomaly. Extensive experiments conducted on several real HSI datasets demonstrate the superiority of SITSR compared with state-of-the-art anomaly detectors.

High-Dimensional Multivariate Linear Regression with Weighted Nuclear Norm Regularization

We consider a low-rank matrix estimation problem when the data is assumed to be generated from the multivariate linear regression model. To induce the low-rank coefficient matrix, we employ the weighted nuclear norm (WNN) penalty defined as the weighted sum of the singular values of the matrix. The weights are set in a nondecreasing order, which yields the non-convexity of the WNN objective function in the parameter space. Although the objective function has been widely applied, studies on the estimation properties of its resulting estimator are limited. We propose an efficient algorithm under the framework of the alternative directional method of multipliers (ADMM) to estimate the coefficient matrix. The estimator from the suggested algorithm converges to a stationary point of an augmented Lagrangian function. Under the orthogonal design setting, the effects of the weights for estimating the singular values of the ground-truth coefficient matrix are derived. Under the Gaussian design setting, a minimax convergence rate on the estimation error is derived. We also propose a generalized cross-validation (GCV) criterion for selecting the tuning parameter and an iterative algorithm for updating the weights. Simulations and a real data analysis demonstrate the competitive performance of our new method. Supplementary materials for this article are available online.

K-nearest-neighbors induced topological PCA for single cell RNA-sequence data analysis

Single-cell RNA sequencing (scRNA-seq) is widely used to reveal heterogeneity in cells, which has given us insights into cell–cell communication, cell differentiation, and differential gene expression. However, analyzing scRNA-seq data is a challenge due to sparsity and the large number of genes involved. Therefore, dimensionality reduction and feature selection are important for removing spurious signals and enhancing downstream analysis. Traditional PCA, a main workhorse in dimensionality reduction, lacks the ability to capture geometrical structure information embedded in the data, and previous graph Laplacian regularizations are limited by the analysis of only a single scale. We propose a topological Principal Components Analysis (tPCA) method by the combination of persistent Laplacian (PL) technique and L2,1 norm regularization to address multiscale and multiclass heterogeneity issues in data. We further introduce a k-Nearest-Neighbor (kNN) persistent Laplacian technique to improve the robustness of our persistent Laplacian method. The proposed kNN-PL is a new algebraic topology technique which addresses the many limitations of the traditional persistent homology. Rather than inducing filtration via the varying of a distance threshold, we introduced kNN-tPCA, where filtrations are achieved by varying the number of neighbors in a kNN network at each step, and find that this framework has significant implications for hyper-parameter tuning. We validate the efficacy of our proposed tPCA and kNN-tPCA methods on 11 diverse benchmark scRNA-seq datasets, and showcase that our methods outperform other unsupervised PCA enhancements from the literature, as well as popular Uniform Manifold Approximation (UMAP), t-Distributed Stochastic Neighbor Embedding (tSNE), and Projection Non-Negative Matrix Factorization (NMF) by significant margins. For example, tPCA provides up to 628%, 78%, and 149% improvements to UMAP, tSNE, and NMF, respectively on classification in the F1 metric, and kNN-tPCA offers 53%, 63%, and 32% improvements to UMAP, tSNE, and NMF, respectively on clustering in the ARI metric.

Polarimetric holographic SAR 3-D imaging based on L 2, 1 -norm regularization: initial result

Polarimetric holographic SAR 3-D imaging based on <i>L</i> _{2, 1} -norm regularization: initial result

Sparse Trace Ratio LDA for Supervised Feature Selection.

Classification is a fundamental task in the field of data mining. Unfortunately, high-dimensional data often degrade the performance of classification. To solve this problem, dimensionality reduction is usually adopted as an essential preprocessing technique, which can be divided into feature extraction and feature selection. Due to the ability to obtain category discrimination, linear discriminant analysis (LDA) is recognized as a classic feature extraction method for classification. Compared with feature extraction, feature selection has plenty of advantages in many applications. If we can integrate the discrimination of LDA and the advantages of feature selection, it is bound to play an important role in the classification of high-dimensional data. Motivated by the idea, we propose a supervised feature selection method for classification. It combines trace ratio LDA with l2,p -norm regularization and imposes the orthogonal constraint on the projection matrix. The learned row-sparse projection matrix can be used to select discriminative features. Then, we present an optimization algorithm to solve the proposed method. Finally, the extensive experiments on both synthetic and real-world datasets indicate the effectiveness of the proposed method.

Prestack sparse envelope seismic inversion method adopting the L0−L2-norm regularization

The high drilling costs limit the possibility of obtaining reliable low-frequency constrained models in marine environments or areas with sparse well-log data, making it difficult to constrain lateral continuity and deep data. To enhance stability and accuracy in prestack seismic inversion in these areas, an elastic parameter estimation approach using sparse envelope inversion with [Formula: see text]-norm regularization is proposed. The method combines signal sparse representation and modulation theories to derive a new formula for sparse envelope extraction at lower frequencies. By applying [Formula: see text]-norm regularization to the sparse envelope, we obtain parameter inversion results with smoothed trends, augmenting low-wavenumber information for improved model constraints. In addition, taking the envelope inversion results obtained by the [Formula: see text] norm as the model constraint, the sparsest inversion results with obvious block-like characteristics are obtained by regularizing the inversion equation with the [Formula: see text] norm. Notably, the new method effectively suppresses wavelet side lobes, resulting in stable and accurate inversions without the need for initial low-frequency models. A synthetic example is used to illustrate the feasibility and stability of proposed approach and further demonstrate its practicality in reservoir parameter estimation through a field case study of gas exploration.

Iterative Regularization with k-support Norm: An Important Complement to Sparse Recovery

Sparse recovery is ubiquitous in machine learning and signal processing. Due to the NP-hard nature of sparse recovery, existing methods are known to suffer either from restrictive (or even unknown) applicability conditions, or high computational cost. Recently, iterative regularization methods have emerged as a promising fast approach because they can achieve sparse recovery in one pass through early stopping, rather than the tedious grid-search used in the traditional methods. However, most of those iterative methods are based on the l1 norm which requires restrictive applicability conditions and could fail in many cases. Therefore, achieving sparse recovery with iterative regularization methods under a wider range of conditions has yet to be further explored. To address this issue, we propose a novel iterative regularization algorithm, IRKSN, based on the k-support norm regularizer rather than the l1 norm. We provide conditions for sparse recovery with IRKSN, and compare them with traditional conditions for recovery with l1 norm regularizers. Additionally, we give an early stopping bound on the model error of IRKSN with explicit constants, achieving the standard linear rate for sparse recovery. Finally, we illustrate the applicability of our algorithm on several experiments, including a support recovery experiment with a correlated design matrix.

Robust treatment planning for small animal radio-neuromodulation using focused kV x-ray beams.

In preclinical radio-neuromodulation research, small animal experiments are pivotal for unraveling radiobiological mechanism, investigating prescription and planning techniques, and assessing treatment effects and toxicities. However, the target size inside a rat brain is typically in the order of sub-millimeters. The small target inside the visual cortex neural region in rat brain with a diameter of around 1mm was focused in this work to observe the physiological change of this region. Delivering uniform doses to the small target while sparing health tissues is challenging. Focused kV x-ray technique based on modern x-ray polycapillary focusing lens is a promising modality for small animal radio-neuromodulation. The current manual planning method could lead to sub-optimal plans, and the positioning uncertainties due to mechanical accuracy limitations, animal immobilization, and robotic arm motion are not considered. This work aims to design a robust inverse planning method to optimize the intensities of focused kV x-ray beams located in beam trajectories to irradiate small mm-sized targets in rat brains for radio-neuromodulation. Focused kV x-ray beams were generated through polycapillary x-ray focusing lenses on achieving small (≤0.3mm) focus perpendicular to the beam. The beam trajectories were manually designed in 3D space in scanning-while-rotating mode. Geant4 Monte Carlo (MC) simulation generated a dose calculation matrix for each focused kV x-ray beam located in beam trajectories. In the proposed robust inverse planning method, an objective function combining a voxel-wise stochastic programming approach and L1 norm regularization was established to overcome the positioning uncertainties and obtain a high-quality plan. The fast iterative shrinkage thresholding algorithm (FISTA) was utilized to solve the objective function and obtain the optimal intensities. Four cases were employed to validate the feasibility and effectiveness of the proposed method. The manual and non-robust inverse planning methods were also implemented for comparison. The proposed robust inverse planning method achieved superior dose homogeneity and higher robustness against positioning uncertainties. On average, the clinical target volume (CTV) homogeneity index (HI) of robust inverse plan improved to 13.3 from 22.9 in non-robust inverse plan and 53.8 in manual plan if positioning uncertainties were also present. The average bandwidth at D90 was reduced by 6.5Gy in the robust inverse plan, compared to 9.6Gy in non-robust inverse plan and 12.5Gy in manual plan. The average bandwidth at D80 was reduced by 3.4Gy in robust inverse plan, compared to 5.5Gy in non-robust inverse plan and 8.5Gy in manual plan. Moreover, the dose delivery time of manual plan was reduced by an average reduction of 54.7% with robust inverse plan and 29.0% with non-robust inverse plan. Compared to manual and non-robust inverse planning methods, the robust inverse planning method improved the dose homogeneity and delivery efficiency and was resistant to the uncertainties, which are crucial for radio-neuromodulation utilizing focused kV x-rays.

Local dual-graph discriminant classifier for binary classification

Graph-based methods mine the potential structural information of data by constructing various graphs that positively affect the classifiers when dealing with classification problems. However, traditional graph-based classifiers are the most common single-graph classifiers and minimize only intra-class compactness, where inter-class separability is replaced by other factors. To consider real inter-class separability, we introduce a novel local dual-graph structure that can fully mine the geometric distribution of data by simultaneously maximizing the inter-class separability and minimizing the intra-class compactness. This local dual-graph structure reflects the relationship between samples and their neighbors and hence avoids the negative impact of outliers on the construction of graphs. Furthermore, a novel classifier called the local dual-graph discriminant classifier (LDGDC) is proposed using a local dual-graph structure. Originally, LDGDC is designed to perform the following optimization: minimization of the 2-norm regularization of model coefficients and intra-class compactness, and maximization of the inter-class separability, which is a non-convex optimization problem. To facilitate the solution, we transform the original non-convex problem of LDGDC into a convex problem. Finally, experiments were conducted on several public datasets, and the results demonstrate the effectiveness and robustness of the proposed LDGDC.

Efficient methods for iterative ultrasound image reconstruction using L1 and L2 norm regularization

Ultrasound image reconstruction can be posed as an inverse problem, with image pixels as parameters to a model of wave propagation that are estimated from raw ultrasound channel data. Such framework allows one to make assumptions about channel signals and image properties in form of regularization that can be used to improve reconstruction accuracy in the presence of electronic and acoustic noise. A simple model assumes spherical wave propagation from point sources in a homogeneous medium, similar to delay-and-sum (DAS) beamforming; solving such problems numerically can require many iterations and can result in blurred images when traditional L2 norm regularization is used. We use Fast Iterative Shrinkage Thersholding Algorithm (FISTA) to reduce the number of iterations to less than 10, and to apply L1-norm regularization to the data, which improves edge sharpness and image contrast. We demonstrate the concept using FIELD II simulated ultrasound signals from point, speckle, and anechoic targets with controlled levels of electronic and speckle noise. The FISTA-reconstructed point targets show reduced sidelobe levels by 15 dB compared to the traditional DAS image, and the reduction in full-width at half maximum by a factor of 2. We also implement iterative reconstruction in omega-k frequency domain, which allows factoring the forward and adjoint operators at each spatial frequency and paves the way for an intuitive and more memory efficient, multi-threaded implementation of the method.