L1-norm Minimization Problem Research Articles

On the atout ticket learning problem for neural networks and its application in securing federated learning exchanges

Artificial Neural Networks (ANNs) have become the backbone of many real-world applications, including distributed applications relying on Federated Learning (FL). However, several vulnerabilities/attacks have emerged in recent years, affecting the benefits of using ANNs in FL, such as reconstruction attacks and membership inference attacks. These attacks can have severe impacts on both the societal and professional levels. For instance, inferring the presence of a patient’s private health record in a medical study or a clinic database violates the patient’s privacy and can have legal or ethical consequences. Therefore, protecting the data and model from malicious attacks in FL systems is important. This paper introduces the Atout Ticket Learning (ATL) problem. This new problem consists of identifying sensitive parameters (atout tickets) of a neural network model, which, if modified, will increase the model’s loss by at least a given threshold ϵ. First, we formulate ATL as an ℓ0-norm minimization problem, and we derive a lower bound on the number of atout tickets needed to achieve a model degradation of ϵ. Second, we design the Atout Ticket Protocol (ATP) as an effective solution for privacy-preserving in FL systems using atout tickets, along with the benefit of noise perturbations and secure aggregation techniques. Finally, we experiment ATP against FL reconstruction attacks using new selection strategies, namely Inverting Gradients, Deep Leakage, and Improved Deep Leakage. The results show that ATP is highly robust against these attacks.

Gridless Evolutionary Approach for Line Spectral Estimation With Unknown Model Order.

Gridless methods show great superiority in line spectral estimation. These methods need to solve an atomic l0 norm (i.e., the continuous analog of l0 norm) minimization problem to estimate frequencies and model order. Since this problem is NP-hard to compute, relaxations of the atomic l0 norm, such as the nuclear norm and reweighted atomic norm, have been employed for promoting sparsity. However, the relaxations give rise to a resolution limit, subsequently leading to biased model order and convergence error. To overcome the above shortcomings of relaxation, we propose a novel idea of simultaneously estimating the frequencies and model order using the atomic l0 norm. To accomplish this idea, we build a multiobjective optimization model. The measurement error and the atomic l0 norm are taken as the two optimization objectives. The proposed model directly exploits the model order via the atomic l0 norm, thus breaking the resolution limit. We further design a variable-length evolutionary algorithm to solve the proposed model, which includes two innovations. One is a variable-length coding and search strategy. It flexibly codes and interactively searches diverse solutions with different model orders. These solutions act as steppingstones that helpfully exploring the variable and open-ended frequency search space and provide extensive potentials toward the optima. Another innovation is a model-order pruning mechanism, which heuristically prunes less contributive frequencies within the solutions, thus significantly enhancing convergence and diversity. Simulation results confirm the superiority of our approach in both frequency estimation and model-order selection.

Distributed continuous-time accelerated neurodynamic approaches for sparse recovery via smooth approximation to [formula omitted]-minimization

This paper develops two continuous-time distributed accelerated neurodynamic approaches for solving sparse recovery via smooth approximation to L1-norm minimization problem. First, the L1-norm minimization problem is converted into a distributed smooth optimization problem by utilizing multiagent consensus theory and smooth approximation. Then, a distributed primal–dual accelerated neurodynamic approach is designed by using Karush–Kuhn–Tucker (KKT) condition and Nesterov’s accelerated method. Furthermore, in order to reduce the structure complexity of the presented neurodynamic approach, based on the projection matrix, we eliminate a dual variable in the KKT condition and propose a distributed accelerated neurodynamic approach with a simpler structure. It is proved that the two proposed distributed neurodynamic approaches both achieve O(1t2) convergence rate. Finally, the simulation results of sparse recovery are given to demonstrate the effectiveness of the proposed approaches.

L0 Gradient-Regularization and Scale Space Representation Model for Cartoon and Texture Decomposition.

In this paper, we consider decomposing an image into its cartoon and texture components. Traditional methods, which mainly rely on the gradient amplitude of images to distinguish between these components, often show limitations in decomposing small-scale, high-contrast texture patterns and large-scale, low-contrast structural components. Specifically, these methods tend to decompose the former to the cartoon image and the latter to the texture image, neglecting the scale features inherent in both components. To overcome these challenges, we introduce a new variational model which incorporates an L0-based total variation norm for the cartoon component and an L2 norm for the scale space representation of the texture component. We show that the texture component has a small L2 norm in the scale space representation. We apply a quadratic penalty function to handle the non-separable L0 norm minimization problem. Numerical experiments are given to illustrate the efficiency and effectiveness of our approach.

A truncated approximate difference algorithm for sparse signal recovery

In this paper, we study the regularization lp-norm minimization problem to recover the sparse signals. We first prove that every global optimal solution to the regularization lp-norm minimization problem also solves the l0-norm minimization problem if the certain conditions are satisfied, and then generate a truncated approximated difference algorithm to recover the sparse signals. At last, we provide some numerical simulations to test the performance of the truncated approximated difference algorithm, and the numerical results show that the proposed algorithm performs effectively in recovering the sparse signals compared with some state-of-art methods.

Semi-Sparsity for Smoothing Filters.

In this paper, we propose a semi-sparsity smoothing method based on a new sparsity-induced minimization scheme. The model is derived from the observations that semi-sparsity prior knowledge is universally applicable in situations where sparsity is not fully admitted such as in the polynomial-smoothing surfaces. We illustrate that such priors can be identified into a generalized L0-norm minimization problem in higher-order gradient domains, giving rise to a new "feature-aware" filter with a powerful simultaneous-fitting ability in both sparse singularities (corners and salient edges) and polynomial-smoothing surfaces. Notice that a direct solver to the proposed model is not available due to the non-convexity and combinatorial nature of L0-norm minimization. Instead, we propose to solve it approximately based on an efficient half-quadratic splitting technique. We demonstrate its versatility and many benefits to a series of signal/image processing and computer vision applications.

Inverse Synthetic Aperture Radar Sparse Imaging Exploiting the Group Dictionary Learning

Sparse imaging relies on sparse representations of the target scenes to be imaged. Predefined dictionaries have long been used to transform radar target scenes into sparse domains, but the performance is limited by the artificially designed or existing transforms, e.g., Fourier transform and wavelet transform, which are not optimal for the target scenes to be sparsified. The dictionary learning (DL) technique has been exploited to obtain sparse transforms optimized jointly with the radar imaging problem. Nevertheless, the DL technique is usually implemented in a manner of patch processing, which ignores the relationship between patches, leading to the omission of some feature information during the learning of the sparse transforms. To capture the feature information of the target scenes more accurately, we adopt image patch group (IPG) instead of patch in DL. The IPG is constructed by the patches with similar structures. DL is performed with respect to each IPG, which is termed as group dictionary learning (GDL). The group oriented sparse representation (GOSR) and target image reconstruction are then jointly optimized by solving a l1 norm minimization problem exploiting GOSR, during which a generalized Gaussian distribution hypothesis of radar image reconstruction error is introduced to make the imaging problem tractable. The imaging results using the real ISAR data show that the GDL-based imaging method outperforms the original DL-based imaging method in both imaging quality and computational speed.

A continuous-time neurodynamic approach and its discretization for distributed convex optimization over multi-agent systems

Distributed optimization problem (DOP) over multi-agent systems, which can be described by minimizing the sum of agents’ local objective functions, has recently attracted widespread attention owing to its applications in diverse domains. In this paper, inspired by penalty method and subgradient descent method, a continuous-time neurodynamic approach is proposed for solving a DOP with inequality and set constraints. The state of continuous-time neurodynamic approach exists globally and converges to an optimal solution of the considered DOP. Comparisons reveal that the proposed neurodynamic approach can not only resolve more general convex DOPs, but also has lower dimension of solution space. Additionally, the discretization of the neurodynamic approach is also introduced for the convenience of implementation in practice. The iteration sequence of discrete-time method is also convergent to an optimal solution of DOP from any initial point. The effectiveness of the neurodynamic approach is verified by simulation examples and an application in L1-norm minimization problem in the end.

Alternating Direction Method of Multipliers for TOA-Based Positioning Under Mixed Sparse LOS/NLOS Environments

For positioning system based on wireless sensor networks, NLOS errors are one of the main factors to degrade localization performance of an algorithm, about which lots of research results and analysis have been published in previous literatures to enhance localization performance. However, those literatures have neglected computational time, another important index to performance. To decrease computational time and improve localization accuracy simultaneously, we firstly consider NLOS errors as outliers and transform the TOA-based localization problem into a sparse optimization one in LOS-dominating environment. Then, we introduce sparse technology into line-of-sight/none-line-of-sight (LOS/NLOS) scenarios formulating a L1-norm minimization problem, and solve it by alternating direction method of multipliers (ADMM) with a strategy of iterative adaptive. Monte Carlo simulation results show that our method has advantages of high computation speed and positioning accuracy under mixed sparse LOS/NLOS scenarios.

3-D Mixed Far-Field and Near-Field Sources Localization With Cross Array

This correspondence proposes a localization method for mixed far-field (FF) and near-field (NF) sources based on cross arrays where two-dimensional (2-D) direction-of-arrival (DOA) and range estimation are considered. For 2-D DOA estimation, we first construct a range-free cumulant-based vector and employ the covariance matching criterion and Vandermonde decomposition theorem to retrieve DOA information. We then build a sparse model with respect to range parameter and solve an l 1 -norm minimization problem to obtain range estimates. Our method enjoys the following advantages: 1) our method can locate mixed FF and NF sources; 2) the azimuth, elevation and range can be automatically paired; 3) array aperture loss can be avoided. Simulation results show the effectiveness of our method.

Cascaded Form Sparse FIR Filter Design

In this paper, a cascaded form sparse finite impulse response (FIR) filter design algorithm is proposed. By decomposing a filter into two subfilters, the sparse filter design problem is reformulated as a quadratic constrained l 0 -norm minimization problem, which is further transformed to a convex problem through the introduction of a binary masking vector. An iterative forth-and-back search with sequence direction and error direction control is proposed to optimize the binary masking vector by minimizing the distance between the minimax approximation error and the specified upper bound. Experimental results show that the proposed design algorithm can improve the sparsity with comparable or even lower effective filter order.

Super-resolved water/fat image reconstruction based on single-shot spatiotemporally encoded MRI

Single-shot spatiotemporally encoded (SPEN) MRI has been validated to possess considerable performance in both spatial and temporal resolution. Water/fat separation is essential for MRI applications in which only water signal is needed. In this article, a super-resolved water/fat image reconstruction method (dubbed SWAF) combined prior knowledge was developed based on single-shot SPEN MRI. The point spread function of spatiotemporal encoding under multiple chemical shifts situation was derived and used for constructing an equation for SWAF image reconstruction. By processing the prior chemical shift information with filtering operation, an initial spin density profile of water/fat and a weighting matrix for water/fat residual artifacts suppression were obtained to guide the reconstruction process. A l1 norm minimization problem with regularization was exploited to reconstruct separated water/fat images with high spatial resolution and less residual/aliasing artifacts. Numeric simulation and experiments on water–oil phantom and rat abdomen/neck imaging demonstrated the effectiveness and robustness of this new method. The SWAF method proposed herein would promote the application of SPEN MRI in the cases where water/fat separation is required.

Fast kernel sparse representation based classification for Undersampling problem in face recognition

We propose a fast kernel sparse representation based classification (SRC) for undersampling problem, i.e., each class has very few training samples, in face recognition. The proposed algorithm exploits a nonlinear mapping to map the data from the original input space into a high-dimensional feature space. Then, it performs very fast sparse representation and classification of samples in this space. Similar to the typical SRC methods, the proposed approach is based on the L1 norm minimization, whose direct solution can be very time-consuming. In order to improve the computational efficiency, our method uses the coordinate descent method in the feature space, which can avoid directly solving the L1 norm minimization problem, and significantly expedites the computational procedure. Compared with other SRC methods based on the L1 norm minimization, our proposed method achieves very high computational efficiency, without significantly degrading the classification performance. Several experiments on popular face databases demonstrate that our method is a promising efficient kernel SRC based method.

Improving performance of medical image fusion using histogram, dictionary learning and sparse representation

Medical image fusion has attracted much attention in recent years, which aims to fuse different medical images into a more informative and clearer one. The fused image is able to help doctors to diagnose diseases rapidly and effectively. Among numerous fusion methods, sparse-representation-based image fusion is a new concept that has emerged over the past several years. However, the high-frequency components of low-resolution and the high-frequency components of source images are obtained equally, and sparse coefficients are solved by a minimization problem. As a result, it ignores the correlation between high-frequency components of low-resolution and the high-frequency components of source images, and solutions to the L0-norm minimization problem. To address these issues, we propose a new image fusion method based on histogram similarity and multi-view weighted sparse representation. By introducing a histogram similarity, different weights are assigned to the high-frequency components of low-resolution and the high-frequency components of source images to efficiently harness the complementary information. In addition, sparse coefficients solved by the L1-norm minimization problem are more accurate. This technique is further incorporated into medical image fusion. Experimental results demonstrate that the proposed method achieves state-of-the-art performance in terms of both visual quality and quantitative evaluation metrics.

2DRLPP: Robust two-dimensional locality preserving projection with regularization

The recently proposed two-dimensional locality preserving projection (2DLPP) is an excellent matrix-based dimensionality reduction method. However, the formulation of 2DLPP may encounter several drawbacks in real-world applications, such as sensitiveness to outliers, non-orthogonal projections, and singularity problem. To alleviate these issues, in this paper, we propose a novel robust two-dimensional locality preserving projection (2DRLPP) for noisy image recognition. The proposed 2DRLPP preserves the local manifold structure of two-dimensional image space under the robust L1-norm criterion. Different from the existing 2DLPP, our 2DRLPP enjoys several incomparable advantages: (i) 2DRLPP extracts projections from matrix-based image space via the L1-norm locality preserving criterion rather than the L2-norm one, and hence it is more robust to outliers. (ii) The orthogonality constraint is further imposed on 2DRLPP to ensure its orthogonal projections. (iii) The introduced regularization term can not only control the model complexity but also guarantee the stability of solution. (iv) A simple but efficient iterative algorithm is presented to solve the corresponding L1-norm minimization problem, whose convergence is guaranteed theoretically. Extensive experimental results on three noisy face image datasets confirm the feasibility and robustness of the proposed approach.

A new piecewise quadratic approximation approach for L0 norm minimization problem

In this paper, we consider the problem of finding sparse solutions for underdetermined systems of linear equations, which can be formulated as a class of $L_{0}$ norm minimization problem. By using the least absolute residual approximation, we propose a new piecewise quadratic function to approximate the $L_{0}$ norm. Then, we develop a piecewise quadratic approximation (PQA) model where the objective function is given by the summation of a smooth non-convex component and a non-smooth convex component. To solve the (PQA) model, we present an algorithm based on the idea of the iterative thresholding algorithm and derive the convergence and the convergence rate. Finally, we carry out a series of numerical experiments to demonstrate the performance of the proposed algorithm for (PQA). We also conduct a phase diagram analysis to further show the superiority of (PQA) over $L_{1}$ and $L_{1/2}$ regularizations.

A Novel Video Coding Framework Using a Self-Adaptive Dictionary

In this paper, we propose to use a self-adaptive redundant dictionary, consisting of all possible inter and intra prediction candidates, to directly represent the frame blocks in a video sequence. The self-adaptive dictionary generalizes the conventional predictive coding approach by allowing adaptive linear combinations of prediction candidates, which is solved by an rate-distortion aware L0-norm minimization problem using orthogonal least squares (OLS). To overcome the inefficiency in quantizing and coding coefficients corresponding to correlated chosen atoms, we orthonormalize the chosen atoms recursively as part of OLS process. We further propose a two-stage video coding framework, in which a second stage codes the residual from the chosen atoms using a modified discrete cosine transform (DCT) dictionary that is adaptively orthonormalized with respect to the subspace spanned by the first stage atoms. To determine the transition from the first stage to the second stage, we propose a rate-distortion (RD) aware adaptive switching algorithm. The proposed framework is further extended to accommodate variable block sizes ( $16\times 16$ , $8\times 8$ , and $4\times 4$ ), and the partition mode is derived by a fast partition mode decision algorithm. A context-adaptive binary arithmetic entropy coder is designed to code the symbols of the proposed coding framework. The proposed coder shows competitive, and in some cases better RD performance, compared with the HEVC video coding standard for P-frames.

An improved locality preserving projection with ℓ1-norm minimization for dimensionality reduction

Abstract Locality preserving projection (LPP) is a classical tool for dimensionality reduction and feature extraction. It usually makes use of the l2-norm criterion for optimization, and is thus sensitive to outliers. In order to achieve robustness, LPP-L1 is proposed by employing the l1-norm as distance criterion. However, the edge weights of LPP-L1 measure only the dissimilarity of pairs of vertices and ignore the preservation of the similarity. In this paper, we develop a novel algorithm, termed as ILPP-L1, in which the l1-norm is utilized to obtain robustness and the similarities of pairs of vertices are effectively preserved, simultaneously. ILPP-L1 is robust to outliers because of the use of the l1-norm. The l1-norm minimization problem is directly solved, which ensures the preservation of the similarity of pairs of vertices. The solution is justified to converge to local minimum. In addition, ILPP-L1 avoids small sample size problem. Experiment results on benchmark databases confirm the effectiveness of the proposed method.

Design of Sparse FIR Decision Feedback Equalizers in MIMO Systems Using Hybrid l₁/l₂ Norm Minimization and the OMP Algorithm.

In this paper, a novel scheme using hybrid l1/l2 norm minimization and the orthogonal matching pursuit (OMP) algorithm is proposed to design the sparse finite impulse response (FIR) decision feedback equalizers (DFE) in multiple input multiple output (MIMO) systems. To reduce the number of nonzero taps for the FIR DFE while ensuring its design accuracy, the problem of designing a sparse FIR DFE is transformed into an l0 norm minimization problem, and then the proposed scheme is used to obtain the sparse solution. In the proposed scheme, a sequence of minimum weighted l2 norm problems is solved using the OMP algorithm. The nonzero taps positions can be corrected with the different weights in the diagonal weighting matrix which is computed through the hybrid l1/l2 norm minimization. The simulation results verify that the sparse FIR MIMO DFEs designed by the proposed scheme get a significant reduction in the number of nonzero taps with a small performance loss compared to the non-sparse optimum DFE under the minimum mean square error (MMSE) criterion. In addition, the proposed scheme provides better design accuracy than the OMP algorithm with the same sparsity level.

Two-layer compressive sensing based video encoding and decoding framework for WMSN

Acquiring massive data from wireless devices such as video sampling and transmission remains a challenge in the resource-restricted wireless multimedia sensor networks (WMSNs). To address the conflict between massive data processing and limited bandwidth or energy, we propose a two-layer compressive sensing based video encoding and decoding framework, which not only significantly reduces the amount of sampling data, but also transfers computation burden from distributed sensor nodes to powerful sink node. To successfully decode compressed data, at the first layer, we construct the first group sparse basis by exploiting the correlation between frames for each sensor node. At the second layer, we consider the spatial correlation between neighbor nodes. By employing dictionary learning, we obtain the second group sparse basis. The sink node employs both groups of sparse bases to perform l1 norm minimization problem twice and recover original video for each node. Experiment results show that, our CS based framework has high compression efficiency due to our constructed sparse basis. Compared with existing sparse basis, our sparse basis provides better sparse representation.