Sparsity Level Research Articles

Channel estimation for reconfigurable intelligent surface-assisted mmWave based on Re‘nyi entropy function

This study focuses on channel estimation for reconfigurable intelligent surface (RIS)-assisted mmWave systems, in which the RIS is used to facilitate base-to-user data transfer. For beamforming to work with active and passive elements, a large-size cascade channel matrix should always be known. Low training costs are achieved by using the mmWave channels’ inherent sparsity. The research provides a unique compressive sensing-based channel estimation approach for reducing pilot overhead issues to a minimum. The proposed technique estimates channel data signals in a downlink for RIS-assisted mmWave systems. The mmWave systems often have a sparse distribution of signal sources due to the spatial correlations of the domains. This distribution pattern makes it possible to use compressive sensing methods to resolve the channel estimation issue. In order to decrease the pilot overhead, which is necessary to predict the channel, the proposed method extends the Re‘nyi entropy function as the sparsity-promoting regularizer. In contrast to conventional compressive sensing techniques, which necessitate an initial knowledge of the signal’s sparsity level, the presented method employs sparsity adaptive matching pursuit (SAMP) techniques to gradually determine the signal’s sparsity level. Furthermore, it introduces a threshold parameter based on the signal’s energy level to eliminate the sparsity level requirement. Extensive simulations show that the presented channel estimation approach surpasses the traditional OMP-based channel estimation methods in terms of normalized mean square error performance. In addition, the computational cost of channel estimation is lowered. Based on the simulations, our approach can estimate the channel well while reducing training overhead by a large amount.

NetSHy: network summarization via a hybrid approach leveraging topological properties.

Biological networks can provide a system-level understanding of underlying processes. In many contexts, networks have a high degree of modularity, i.e. they consist of subsets of nodes, often known as subnetworks or modules, which are highly interconnected and may perform separate functions. In order to perform subsequent analyses to investigate the association between the identified module and a variable of interest, a module summarization, that best explains the module's information and reduces dimensionality is often needed. Conventional approaches for obtaining network representation typically rely only on the profiles of the nodes within the network while disregarding the inherent network topological information. In this article, we propose NetSHy, a hybrid approach which is capable of reducing the dimension of a network while incorporating topological properties to aid the interpretation of the downstream analyses. In particular, NetSHy applies principal component analysis (PCA) on a combination of the node profiles and the well-known Laplacian matrix derived directly from the network similarity matrix to extract a summarization at a subject level. Simulation scenarios based on random and empirical networks at varying network sizes and sparsity levels show that NetSHy outperforms the conventional PCA approach applied directly on node profiles, in terms of recovering the true correlation with a phenotype of interest and maintaining a higher amount of explained variation in the data when networks are relatively sparse. The robustness of NetSHy is also demonstrated by a more consistent correlation with the observed phenotype as the sample size decreases. Lastly, a genome-wide association study is performed as an application of a downstream analysis, where NetSHy summarization scores on the biological networks identify more significant single nucleotide polymorphisms than the conventional network representation. R code implementation of NetSHy is available at https://github.com/thaovu1/NetSHy. Supplementary data are available at Bioinformatics online.

A new image decomposition approach using pixel-wise analysis sparsity model

Decomposing an image into two ‘simpler’ layers has been widely used in low-level vision tasks, such as image recovery and enhancement. It is an ill-posed problem since the number of unknowns are larger than the input. In this paper, a two-step strategy is introduced, including task-aware priors estimate and a decomposition model. A pixel-wise analysis sparsity model is proposed to regularize the separation layers, which supposes the transformed image generated with analysis operator is sparse. Unlike regularizing all pixels with one penalty weight, we try to estimate each pixel’s sparsity level with task-aware priors and to achieve pixel-wise sparse penalty. Additionally, one separation layer is regularized with both synthesis sparsity model and pixel-wise analysis sparsity model to exploit their complementary mechanisms. Unlike the analysis one utilizing image local features, the synthesis one exploits an over-complete dictionary and non-local similarity cues to provide flexible prior for regularizing the decomposition results. The proposed model is solved by an alternating optimization algorithm. We evaluate it with two applications, Retinex model and rain streaks removal. Extensive experiments on multiple enhancement datasets, many synthetic and real rainy images demonstrate that our method can remove imaging noise during Retinex decomposition, and can produce high fidelity deraining results. It achieves competing performance in terms of quantitative metrics and visual quality compared with the state-of-the-art methods.

Penalized logistic regression with prior information for microarray gene expression classification.

Cancer classification and gene selection are important applications in DNA microarray gene expression data analysis. Since DNA microarray data suffers from the high-dimensionality problem, automatic gene selection methods are used to enhance the classification performance of expert classifier systems. In this paper, a new penalized logistic regression method that performs simultaneous gene coefficient estimation and variable selection in DNA microarray data is discussed. The method employs prior information about the gene coefficients to improve the classification accuracy of the underlying model. The coordinate descent algorithm with screening rules is given to obtain the gene coefficient estimates of the proposed method efficiently. The performance of the method is examined on five high-dimensional cancer classification datasets using the area under the curve, the number of selected genes, misclassification rate and F-score measures. The real data analysis results indicate that the proposed method achieves a good cancer classification performance with a small misclassification rate, large area under the curve and F-score by trading off some sparsity level of the underlying model. Hence, the proposed method can be seen as a reliable penalized logistic regression method in the scope of high-dimensional cancer classification.

Dynamic Characteristics Preserving Data Compressing Algorithm for Transactive Energy Management Frameworks

Several players are enabled in smart grids and involved in decision-making process. These players are fully interconnected in the transactive energy (TE) framework to operate the system optimally. By implementing the TE, the final decision are adopted after various consensus between participants. Hence, huge number of data and information are communicated in this framework which leads to complex communication algorithms and wide-bandwidth channels. Besides, the prosumers dataset in the TE framework are with low level of sparsity which is due to their independent interactions with peers and the grid. This article proves the low sparsity in the prosumers dataset and shows the shortcomings of the existing methods. A dynamic intelligent algorithm is proposed in this article to characterize the prosumers data based on the mutual information (MI) theorem. In addition, two data compression algorithms have been proposed in this article to reduce the bandwidth and space for communicating and storing purposes, respectively. The proposed algorithms for data modeling and compression provide compatible superlative performance with minimum information loss. This article saves more communication channel bandwidth by comparing the conventional methods. Fast and robust adaptation of the proposed algorithm facilitates the practical implementation of energy management in the TE framework when wide data transmission is needed. The performance of the proposed algorithms has been evaluated using simulated prosumers dataset and real-world residential and industrial consumer data.

Group-Wise Shrinkage Estimation in Penalized Model-Based Clustering

Finite Gaussian mixture models provide a powerful and widely employed probabilistic approach for clustering multivariate continuous data. However, the practical usefulness of these models is jeopardized in high-dimensional spaces, where they tend to be over-parameterized. As a consequence, different solutions have been proposed, often relying on matrix decompositions or variable selection strategies. Recently, a methodological link between Gaussian graphical models and finite mixtures has been established, paving the way for penalized model-based clustering in the presence of large precision matrices. Notwithstanding, current methodologies implicitly assume similar levels of sparsity across the classes, not accounting for different degrees of association between the variables across groups. We overcome this limitation by deriving group-wise penalty factors, which automatically enforce under or over-connectivity in the estimated graphs. The approach is entirely data-driven and does not require additional hyper-parameter specification. Analyses on synthetic and real data showcase the validity of our proposal.

QLP: Deep Q-Learning for Pruning Deep Neural Networks

We present a novel, deep Q-learning based method, QLP, for pruning deep neural networks (DNNs). Given a DNN, our method intelligently determines favorable layer-wise sparsity ratios, which are then implemented via unstructured, magnitude-based, weight pruning. In contrast to previous reinforcement learning (RL) based pruning methods, our method is not forced to prune a DNN within a single, sequential pass from the first layer to the last. It visits each layer multiple times and prunes them little by little at each visit, achieving superior granular pruning. Moreover, our method is not restricted to a subset of actions within the feasible action space. It has the flexibility to execute a whole range of sparsity ratios (0% - 100%) for each layer. This enables aggressive pruning without compromising accuracy. Furthermore, our method does not require a complex state definition; it features a simple, generic definition that is composed of only the index and the density of the layers, which leads to less computational demand while observing the state at each interaction. Lastly, our method utilizes a carefully designed curriculum that enables learning targeted policies for each sparsity regime, which helps to deliver better accuracy, especially at high sparsity levels. We conduct batched performance tests at compelling sparsity levels (up to 98%), present extensive ablation studies to justify our RL-related design choices, and compare our method with the state-of-the-art, including RL-based and other pruning methods. Our method sets the new state-of-the-art results in most of the experiments with ResNet-32 and ResNet-56 over CIFAR-10 dataset as well as ResNet-50 and MobileNet-v1 over ILSVRC2012 (ImageNet) dataset.

On-grid compressive sampling for spherical field measurements in acoustics.

We derive a compressive sampling method for acoustic field reconstruction using field measurements on a pre-defined spherical grid that has theoretically guaranteed relations between signal sparsity, measurement number, and reconstruction accuracy. This method can be used to reconstruct band limited spherical harmonic or Wigner D-function series (spherical harmonic series are a special case) with sparse coefficients. Contrasting typical compressive sampling methods for Wigner D-function series that use arbitrary random measurements, the new method samples randomly on an equiangular grid, a practical and commonly used sampling pattern. Using its periodic extension, we transform the reconstruction of a Wigner D-function series into a multi-dimensional Fourier domain reconstruction problem. We establish that this transformation has a bounded effect on sparsity level and provide numerical studies of this effect. We also compare the reconstruction performance of the new approach to classical Nyquist sampling and existing compressive sampling methods. In our tests, the new compressive sampling approach performs comparably to other guaranteed compressive sampling approaches and needs a fraction of the measurements dictated by the Nyquist sampling theorem. Moreover, using one-third of the measurements or less, the new compressive sampling method can provide over 20 dB better de-noising capability than oversampling with classical Fourier theory.

A W-Band Two-Dimensional Monopulse Sparse Array Antenna

A novel W-band 2-D monopulse sparse array antenna with suppressed sidelobe levels is proposed in this article. The 2-D monopulse antenna is composed of a radiation sparse array, an irregular interconnected waveguide -based network, a power divider network, and a sum–difference network. The sparse array uses a box-horn as the radiation element and is optimized based on the genetic algorithm, which exhibits low sparsity and low sidelobe levels. The whole aperture of the antenna is divided into four subarrays. To feed all the radiating elements that are randomly distributed in the subarrays with the same amplitude and equal phase, an irregular interconnected waveguide -based network and a power divider network are designed. Finally, a sum–difference network is connected to the end of the four subarrays to obtain the sum and difference radiation performance results for the monopulse antenna. A 2-D monopulse sparse array antenna with 32 radiating elements is fabricated and measured. The measured <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\vert \text {S}_{\mathbf {11}}\vert \le -10$ </tex-math></inline-formula> dB bandwidth of the antenna is over 6 GHz (from 90 to 96 GHz). At the center frequency, the maximal gain of the sum radiation pattern is approximately 24.0 dBi, with low sidelobe levels (both E-plane and H-plane) less than −19.0 dB. In addition, the null depth of the difference radiation pattern is less than −30.0 dB.

A Unified Programmable Edge Matrix Processor for Deep Neural Networks and Matrix Algebra

Matrix Algebra and Deep Neural Networks represent foundational classes of computational algorithms across multiple emerging applications like Augmented Reality or Virtual Reality, autonomous navigation (cars, drones, robots), data science, and various artificial intelligence-driven solutions. An accelerator-based architecture can provide performance and energy efficiency supporting fixed functions through customized data paths. However, constrained Edge systems requiring multiple applications and diverse matrix operations to be efficiently supported, cannot afford numerous custom accelerators. In this article, we present MxCore, a unified architecture that comprises tightly coupled vector and programmable cores sharing data through highly optimized interconnects along with a configurable hardware scheduler managing the co-execution. We submit MxCore as the generalized approach to facilitate the flexible acceleration of multiple Matrix Algebra and Deep-learning applications across a range of sparsity levels. Unified compute resources improve overall resource utilization and performance per unit area. Aggressive and novel microarchitecture techniques along with block-level sparsity support optimize compute and data-reuse to minimize bandwidth and power requirements enabling ultra-low latency applications for low-power and cost-sensitive Edge deployments. MxCore requires a small silicon footprint of 0.2068 mm 2 , in a modern 7-nm process at 1 GHz and achieves (0.15 FP32 and 0.62 INT8) TMAC/mm 2 , dissipating only 11.66 μW of leakage power. At iso-technology and iso-frequency, MxCore provides an energy efficiency of 651.4×, 159.9×, 104.8×, and 124.2× as compared to the 128-core Nvidia’s Maxwell GPU for dense General Matrix Multiply, sparse Deep Neural Network, Cholesky decomposition, and triangular matrix solve respectively.

Estimation of block sparsity in compressive sensing

Explicitly using the block structure of the unknown signal can achieve better reconstruction performance in compressive sensing. An unknown signal with block structure can be accurately recovered from under-determined linear measurements provided that it is sufficiently block sparse. However, in practice, the block sparsity level is typically unknown. In this paper, we propose a soft measure of block sparsity [Formula: see text] with [Formula: see text], and present a procedure to estimate it by using multivariate centered isotropic symmetric [Formula: see text]-stable random projections. The limiting distribution of the estimator is given. Simulations are conducted to illustrate our theoretical results.

A communication-efficient method for generalized linear regression with ℓ0 regularization

In this paper, based on the Karush–Kuhn–Tucker (KKT) conditions of regularization, we propose a communication-efficient distributed learning approach for high-dimensional and sparse generalized linear models with massive data sets stored across different machines. This proposed method is a support detection and root finding method for generalized linear models in a distributed form. In each round of the proposed method, the support set is first determined by the primal and dual information reduced to the master machine, then the reduced maximum likelihood estimator is obtained by the gradient descent method, among which it only suffices to calculate the gradient vectors on each machine and communicate them instead of the data. We give the optimal -norm error bound for the sequences generated by the proposed algorithm and show that this -norm error bound decays exponentially to the optimal order. Moreover, we show that the oracle estimator can be recovered if the target signal is not less than the detectable level. In addition, an adaptive version of the proposed algorithm is developed to estimate the sparsity level. Simulation studies illustrate the superior performance of the proposed methods.

The impact of sparsity in low-rank recurrent neural networks.

Neural population dynamics are often highly coordinated, allowing task-related computations to be understood as neural trajectories through low-dimensional subspaces. How the network connectivity and input structure give rise to such activity can be investigated with the aid of low-rank recurrent neural networks, a recently-developed class of computational models which offer a rich theoretical framework linking the underlying connectivity structure to emergent low-dimensional dynamics. This framework has so far relied on the assumption of all-to-all connectivity, yet cortical networks are known to be highly sparse. Here we investigate the dynamics of low-rank recurrent networks in which the connections are randomly sparsified, which makes the network connectivity formally full-rank. We first analyse the impact of sparsity on the eigenvalue spectrum of low-rank connectivity matrices, and use this to examine the implications for the dynamics. We find that in the presence of sparsity, the eigenspectra in the complex plane consist of a continuous bulk and isolated outliers, a form analogous to the eigenspectra of connectivity matrices composed of a low-rank and a full-rank random component. This analogy allows us to characterise distinct dynamical regimes of the sparsified low-rank network as a function of key network parameters. Altogether, we find that the low-dimensional dynamics induced by low-rank connectivity structure are preserved even at high levels of sparsity, and can therefore support rich and robust computations even in networks sparsified to a biologically-realistic extent.

Neural Collective Matrix Factorization for integrated analysis of heterogeneous biomedical data.

In many biomedical studies, there arises the need to integrate data from multiple directly or indirectly related sources. Collective matrix factorization (CMF) and its variants are models designed to collectively learn from arbitrary collections of matrices. The latent factors learnt are rich integrative representations that can be used in downstream tasks, such as clustering or relation prediction with standard machine-learning models. Previous CMF-based methods have numerous modeling limitations. They do not adequately capture complex non-linear interactions and do not explicitly model varying sparsity and noise levels in the inputs, and some cannot model inputs with multiple datatypes. These inadequacies limit their use on many biomedical datasets. To address these limitations, we develop Neural Collective Matrix Factorization (NCMF), the first fully neural approach to CMF. We evaluate NCMF on relation prediction tasks of gene-disease association prediction and adverse drug event prediction, using multiple datasets. In each case, data are obtained from heterogeneous publicly available databases and used to learn representations to build predictive models. NCMF is found to outperform previous CMF-based methods and several state-of-the-art graph embedding methods for representation learning in our experiments. Our experiments illustrate the versatility and efficacy of NCMF in representation learning for seamless integration of heterogeneous data. https://github.com/ajayago/NCMF_bioinformatics. Supplementary data are available at Bioinformatics online.

Learning Low-Precision Structured Subnetworks Using Joint Layerwise Channel Pruning and Uniform Quantization

Pruning and quantization are core techniques used to reduce the inference costs of deep neural networks. Among the state-of-the-art pruning techniques, magnitude-based pruning algorithms have demonstrated consistent success in the reduction of both weight and feature map complexity. However, we find that existing measures of neuron (or channel) importance estimation used for such pruning procedures have at least one of two limitations: (1) failure to consider the interdependence between successive layers; and/or (2) performing the estimation in a parametric setting or by using distributional assumptions on the feature maps. In this work, we demonstrate that the importance rankings of the output neurons of a given layer strongly depend on the sparsity level of the preceding layer, and therefore, naïvely estimating neuron importance to drive magnitude-based pruning will lead to suboptimal performance. Informed by this observation, we propose a purely data-driven nonparametric, magnitude-based channel pruning strategy that works in a greedy manner based on the activations of the previous sparsified layer. We demonstrate that our proposed method works effectively in combination with statistics-based quantization techniques to generate low precision structured subnetworks that can be efficiently accelerated by hardware platforms such as GPUs and FPGAs. Using our proposed algorithms, we demonstrate increased performance per memory footprint over existing solutions across a range of discriminative and generative networks.

Nonregular and minimax estimation of individualized thresholds in high dimension with binary responses

Given a large number of covariates Z, we consider the estimation of a high-dimensional parameter θ in an individualized linear threshold θTZ for a continuous variable X, which minimizes the disagreement between sign(X−θTZ) and a binary response Y. While the problem can be formulated into the M-estimation framework, minimizing the corresponding empirical risk function is computationally intractable due to discontinuity of the sign function. Moreover, estimating θ even in the fixed-dimensional setting is known as a nonregular problem leading to nonstandard asymptotic theory. To tackle the computational and theoretical challenges in the estimation of the high-dimensional parameter θ, we propose an empirical risk minimization approach based on a regularized smoothed non-convex loss function. The Fisher consistency of the proposed method is guaranteed as the bandwidth of the smoothed loss is shrunk to 0. Statistically, we show that the finite sample error bound for estimating θ in ℓ2 norm is (slogd/n)β/(2β+1), where d is the dimension of θ, s is the sparsity level, n is the sample size and β is the smoothness of the conditional density of X given the response Y and the covariates Z. The convergence rate is nonstandard and slower than that in the classical Lasso problems. Furthermore, we prove that the resulting estimator is minimax rate optimal up to a logarithmic factor. The Lepski’s method is developed to achieve the adaption to the unknown sparsity s or smoothness β. Computationally, an efficient path-following algorithm is proposed to compute the solution path. We show that this algorithm achieves geometric rate of convergence for computing the whole path. Finally, we evaluate the finite sample performance of the proposed estimator in simulation studies and a real data analysis from the ChAMP (Chondral Lesions And Meniscus Procedures) Trial.

The Price of Competition: Effect Size Heterogeneity Matters in High Dimensions

In high-dimensional sparse regression, would increasing the signal-to-noise ratio while fixing the sparsity level always lead to better model selection? For high-dimensional sparse regression problems, surprisingly, in this paper we answer this question in the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">negative</i> in the regime of linear sparsity for the Lasso method, relying on a new concept we term <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">effect size heterogeneity</i> . Roughly speaking, a regression coefficient vector has high effect size heterogeneity if its nonzero entries have significantly different magnitudes. From the viewpoint of this new measure, we prove that the false and true positive rates achieve the optimal trade-off <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">uniformly</i> along the Lasso path when this measure is maximal in a certain sense, and the worst trade-off is achieved when it is minimal in the sense that all nonzero effect sizes are roughly equal. Moreover, we demonstrate that the first false selection occurs much earlier when effect size heterogeneity is minimal than when it is maximal. The underlying cause of these two phenomena is, metaphorically speaking, the “competition” among variables with effect sizes of the same magnitude in entering the model. Taken together, our findings suggest that effect size heterogeneity shall serve as an important complementary measure to the sparsity of regression coefficients in the analysis of high-dimensional regression problems. Our proofs use techniques from approximate message passing theory as well as a novel technique for estimating the rank of the first false variable.

Multivariate Time Series Imputation: An Approach Based on Dictionary Learning.

The problem addressed by dictionary learning (DL) is the representation of data as a sparse linear combination of columns of a matrix called dictionary. Both the dictionary and the sparse representations are learned from the data. We show how DL can be employed in the imputation of multivariate time series. We use a structured dictionary, which is comprised of one block for each time series and a common block for all the time series. The size of each block and the sparsity level of the representation are selected by using information theoretic criteria. The objective function used in learning is designed to minimize either the sum of the squared errors or the sum of the magnitudes of the errors. We propose dimensionality reduction techniques for the case of high-dimensional time series. For demonstrating how the new algorithms can be used in practical applications, we conduct a large set of experiments on five real-life data sets. The missing data (MD) are simulated according to various scenarios where both the percentage of MD and the length of the sequences of MD are considered. This allows us to identify the situations in which the novel DL-based methods are superior to the existing methods.

Physiological characterization of electrodermal activity enables scalable near real-time autonomic nervous system activation inference.

Electrodermal activities (EDA) are any electrical phxenomena observed on the skin. Skin conductance (SC), a measure of EDA, shows fluctuations due to autonomic nervous system (ANS) activation induced sweat secretion. Since it can capture psychophysiological information, there is a significant rise in the research work for tracking mental and physiological health with EDA. However, the current state-of-the-art lacks a physiologically motivated approach for real-time inference of ANS activation from EDA. Therefore, firstly, we propose a comprehensive model for the SC dynamics. The proposed model is a 3D state-space representation of the direct secretion of sweat via pore opening and diffusion followed by corresponding evaporation and reabsorption. As the input to the model, we consider a sparse signal representing the ANS activation that causes the sweat glands to produce sweat. Secondly, we derive a scalable fixed-interval smoother-based sparse recovery approach utilizing the proposed comprehensive model to infer the ANS activation enabling edge computation. We incorporate a generalized-cross-validation to tune the sparsity level. Finally, we propose an Expectation-Maximization based deconvolution approach for learning the model parameters during the ANS activation inference. For evaluation, we utilize a dataset with 26 participants, and the results show that our comprehensive state-space model can successfully describe the SC variations with high scalability, showing the feasibility of real-time applications. Results validate that our physiology-motivated state-space model can comprehensively explain the EDA and outperforms all previous approaches. Our findings introduce a whole new perspective and have a broader impact on the standard practices of EDA analysis.

Sparse regularization with the ℓ0 norm

We consider a minimization problem whose objective function is the sum of a fidelity term, not necessarily convex, and a regularization term defined by a positive regularization parameter [Formula: see text] multiple of the [Formula: see text] norm composed with a linear transform. This problem has wide applications in compressed sensing, sparse machine learning and image reconstruction. The goal of this paper is to understand what choices of the regularization parameter can dictate the level of sparsity under the transform for a global minimizer of the resulting regularized objective function. This is a critical issue but it has been left unaddressed. We address it from a geometric viewpoint with which the sparsity partition of the image space of the transform is introduced. Choices of the regularization parameter are specified to ensure that a global minimizer of the corresponding regularized objective function achieves a prescribed level of sparsity under the transform. Results are obtained for the spacial sparsity case in which the transform is the identity map, a case that covers several applications of practical importance, including machine learning, image/signal processing and medical image reconstruction.