Sparsity Level Research Articles

DSGI‐Net: Density‐based Selective Grouping Point Cloud Learning Network for Indoor Scene

AbstractIndoor scene point clouds exhibit diverse distributions and varying levels of sparsity, characterized by more intricate geometry and occlusion compared to outdoor scenes or individual objects. Despite recent advancements in 3D point cloud analysis introducing various network architectures, there remains a lack of frameworks tailored to the unique attributes of indoor scenarios. To address this, we propose DSGI‐Net, a novel indoor scene point cloud learning network that can be integrated into existing models. The key innovation of this work is selectively grouping more informative neighbor points in sparse regions and promoting semantic consistency of the local area where different instances are in proximity but belong to distinct categories. Furthermore, our method encodes both semantic and spatial relationships between points in local regions to reduce the loss of local geometric details. Extensive experiments on the ScanNetv2, SUN RGB‐D, and S3DIS indoor scene benchmarks demonstrate that our method is straightforward yet effective.

Variational Bayesian EM Algorithm for Quantile Regression in Linear Mixed Effects Models

This paper extends the normal-beta prime (NBP) prior to Bayesian quantile regression in linear mixed effects models and conducts Bayesian variable selection for the fixed effects of the model. The choice of hyperparameters in the NBP prior is crucial, and we employed the Variational Bayesian Expectation–Maximization (VBEM) for model estimation and variable selection. The Gibbs sampling algorithm is a commonly used Bayesian method, and it can also be combined with the EM algorithm, denoted as GBEM. The results from our simulation and real data analysis demonstrate that both the VBEM and GBEM algorithms provide robust estimates for the hyperparameters in the NBP prior, reflecting the sparsity level of the true model. The VBEM and GBEM algorithms exhibit comparable accuracy and can effectively select important explanatory variables. The VBEM algorithm stands out in terms of computational efficiency, significantly reducing the time and resource consumption in the Bayesian analysis of high-dimensional, longitudinal data.

Analyses of the tail-[formula omitted] minimization for fast and enhanced sparse selections

We investigate the effectiveness and efficiency of the iterative tail-ℓ2 minimization (tail-ℓ2-min) technique for its sparse selection capabilities. We conduct profile analyses on the tail-ℓ2-min, establishing the equivalence of the tail-ℓ2-min problem to a two-stage profile ℓ2 formulation, both featuring analytical solutions. The tail null space property (NSP) of sensing matrix A is shown to be equivalent to the NSP of the newly defined profile matrix Ã. Besides the error bound analysis for the tail-ℓ2-min under the typical tail-NSP condition, a novel error bound of the tail-ℓ2-min formulation is also established without relying on NSP or restricted isometry property (RIP) assumptions. It merely contains tractable coefficients of A, and offers insights into successful recovery, with the observation of the convergent iterative procedure. Numerical studies and the applications to image reconstruction demonstrate the superiority and fast convergence of the tail-ℓ2 sparse solution over state-of-the-art sparse selection methodologies. The sparsity level of a signal that the tail-ℓ2 profile algorithm guarantees the recovery is around 41% higher than that of the basis pursuit algorithm. The analytical solutions of the tail-ℓ2 method at each iteration also ensure that the tail-ℓ2 sparse recovery process is notably fast, especially for high dimensions and high sparsity levels.

Comparison of Reservoir Computing topologies using the Recurrent Kernel approach

Reservoir Computing (RC) has become popular in recent years thanks to its fast and efficient computational capabilities. Standard RC has been shown to be equivalent in the asymptotic limit to Recurrent Kernels, which helps in analyzing its expressive power. However, many well-established RC paradigms, such as Leaky RC, Sparse RC, and Deep RC, are yet to be systematically analyzed in such a way. We define the Recurrent Kernel limit of all these RC topologies and conduct a convergence study for a wide range of activation functions and hyperparameters. Our findings provide new insights into various aspects of Reservoir Computing. First, we demonstrate that there is an optimal sparsity level which grows with the reservoir size. Furthermore, our analysis suggests that Deep RC should use reservoir layers of decreasing sizes. Finally, we perform a benchmark demonstrating the efficiency of Structured Reservoir Computing compared to vanilla and Sparse Reservoir Computing.

Neural architecture search for adversarial robustness via learnable pruning

The convincing performances of deep neural networks (DNNs) can be degraded tremendously under malicious samples, known as adversarial examples. Besides, with the widespread edge platforms, it is essential to reduce the DNN model size for efficient deployment on resource-limited edge devices. To achieve both adversarial robustness and model sparsity, we propose a robustness-aware search framework, an Adversarial Neural Architecture Search by the Pruning policy (ANAS-P). The layer-wise width is searched automatically via the binary convolutional mask, titled Depth-wise Differentiable Binary Convolutional indicator (D2BC). By conducting comprehensive experiments on three classification data sets (CIFAR-10, CIFAR-100, and Tiny-ImageNet) utilizing two adversarial losses TRADES (TRadeoff-inspired Adversarial DEfense via Surrogate-loss minimization) and MART (Misclassification Aware adveRsarial Training), we empirically demonstrate the effectiveness of ANAS in terms of clean accuracy and adversarial robust accuracy across various sparsity levels. Our proposed approach, ANAS-P, outperforms previous representative methods, especially in high-sparsity settings, with significant improvements.

A Novel Topology Adaptation Strategy for Dynamic Sparse Training in Deep Reinforcement Learning.

Deep reinforcement learning (DRL) has been widely adopted in various applications, yet it faces practical limitations due to high storage and computational demands. Dynamic sparse training (DST) has recently emerged as a prominent approach to reduce these demands during training and inference phases, but existing DST methods achieve high sparsity levels by sacrificing policy performance as they rely on the absolute magnitude of connections for pruning and randomly generating connections. Addressing this, our study presents a generic method that can be seamlessly integrated into existing DST methods in DRL to enhance their policy performance while preserving their sparsity levels. Specifically, we develop a novel method for calculating the importance of connections within the model. Subsequently, we dynamically adjust the sparse network topology by dropping existing connections and introducing new connections based on their respective importance values. Through validation on eight widely used simulation tasks, our method improves two state-of-the-art (SOTA) DST approaches by up to 70% in episode return and average return across all episodes under various sparsity levels.

Leveraging discriminative data: A pathway to high-performance, stable One-shot Network Pruning at Initialization

One-shot Network Pruning at Initialization (OPaI) is acknowledged as a highly cost-effective strategy for network pruning. However, it has been observed that OPaI models tend to suffer from reduced accuracy stability as target sparsity increases. This study introduces a novel approach by incorporating Discriminative Data (DD) into OPaI, significantly improving performance at higher sparsity levels while maintaining the “one-shot” nature. Our approach achieves state-of-the-art (SOTA) performance, challenging the previously held belief of OPaI’s data independence. Through detailed ablation studies, we thoroughly investigate the influence of data on OPaI, particularly focusing on how DD addresses a common failure in OPaI known as “layer collapse”. Furthermore, our experiments demonstrate that leveraging DD from various pre-trained models can markedly boost pruning performance across different models without requiring changes to the existing model architectures or pruning methodologies. These significant improvements highlight our method’s high generalizability and stability, paving new paths for advancing pruning strategies. Our code is publicly available at: https://github.com/Nonac/DDOPaI.

Sqrt-Loglogish CNN and Markov model for 5G spectrum sensing application

The research presents innovative methods for spectrum sensing in 5G networks, using the Sqrt-Loglogish convolutional neural network (SL-CNN) and hidden orthogonal intuitionistic fuzzy Markov model (HOIFMM). The proposed methods aim to tackle issues related to detecting principal user signals accurately, mitigating interference, and efficiently utilizing the spectrum in wideband spectrum environments due to their diverse and ever changing characteristics. The Sqrt-Loglogish CNN improves spectrum sensing by addressing static threshold dependency and potential overfitting. The HOIFMM offers a complex framework for predicting sparsity levels and primary user patterns. The results highlight the effectiveness of the suggested techniques in differentiating primary user signals from noise and interference, resulting in enhanced interference management tactics and overall network performance. MATLAB simulation is performed and compared the proposed methods performance with existing state-of-the-art methods such as CNN, deep neural network (DNN), long short-term memory (LSTM) and artificial neural networks (ANN). The proposed method has outperformed existing methods in terms of sensitivity, accuracy, and precision. Future endeavors include improving these methods, investigating sophisticated machine learning algorithms, and doing real world validations to guarantee scalability and resilience in various 5G deployment situations. This research advances the spectrum sensing capabilities in 5G networks, potentially improving efficiency, reliability, and quality of service.

On learning sparse linear models from cross samples

The sample complexity of a sparse linear model where samples are dependent is studied in this paper. We consider a specific dependency structure of the samples which arises in some experimental designs such as drug sensitivity studies, where two sets of objects (drugs and cells) are sampled independently, and after crossing (making all possible combinations of drugs and cells), the resulting output (efficacy of drugs) is measured. We call these types of samples as “cross samples”. The dependency among such samples is strong, and existing theoretical studies are either inapplicable or fail to provide realistic bounds. We aim at analyzing the performance of the Lasso estimator where the underlying distributions are mixtures of Gaussians and the data dependency arises from the crossing procedure. Our theoretical results show that the performance of the Lasso estimator in case of cross samples follows that of the i.i.d. samples with differences in constant factors. Through numerical results, we observe a phase transition: When datasets are too small, the error for cross samples is much larger than for i.i.d. samples, but once the size is large enough, cross samples are nearly as useful as i.i.d. samples. Our theoretical analysis suggests that the transition threshold is governed by the level of sparsity of the true parameter vector being estimated.

Robust sparse recovery with sparse Bernoulli matrices via expanders

Sparse binary matrices are of great interest in the field of sparse recovery, nonnegative compressed sensing, statistics in networks, and theoretical computer science. This class of matrices makes it possible to perform signal recovery with lower storage costs and faster decoding algorithms. In particular, Bernoulli (p) matrices formed by independent identically distributed (i.i.d.) Bernoulli (p) random variables are of practical relevance in the context of noise-blind recovery in nonnegative compressed sensing.In this work, we investigate the robust nullspace property of Bernoulli (p) matrices. Previous results in the literature establish that such matrices can accurately recover n-dimensional s-sparse vectors with m=O(sc(p)log⁡ens) measurements, where c(p)≤p is a constant dependent only on the parameter p. These results suggest that in the sparse regime, as p approaches zero, the (sparse) Bernoulli (p) matrix requires significantly more measurements than the minimal necessary, as achieved by standard isotropic subgaussian designs. However, we show that this is not the case.Our main result characterizes, for a wide range of sparsity levels s, the smallest p for which sparse recovery can be achieved with the minimal number of measurements. We also provide matching lower bounds to establish the optimality of our results and explore connections with the theory of invertibility of discrete random matrices and integer compressed sensing.

Optimal Network Pairwise Comparison

We are interested in the problem of two-sample network hypothesis testing: given two networks with the same set of nodes, we wish to test whether the underlying Bernoulli probability matrices of the two networks are the same or not. We propose Interlacing Balance Measure (IBM) as a new two-sample testing approach. We consider the Degree-Corrected Mixed-Membership (DCMM) model for undirected networks, where we allow severe degree heterogeneity, mixed-memberships, flexible sparsity levels, and weak signals. In such a broad setting, how to find a test that has a tractable limiting null and optimal testing performances is a challenging problem. We show that IBM is such a test: in a broad DCMM setting with only mild regularity conditions, IBM has N ( 0 , 1 ) as the limiting null and achieves the optimal phase transition. While the above is for undirected networks, IBM is a unified approach and is directly implementable for directed networks. For a broad directed-DCMM (extension of DCMM for directed networks) setting, we show that IBM has N ( 0 , 1 / 2 ) as the limiting null and continues to achieve the optimal phase transition. We have also applied IBM to the Enron email network and a gene co-expression network, with interesting results.

Improved gradient leakage attack against compressed gradients in federated learning

Distributed machine learning, such as federated learning, protects privacy by collecting gradients instead of training data. Recent studies have shown that gradient leakage attacks are possible in distributed machine learning, that is, the training data can be reconstructed from the shared gradients. In practical applications, distributed machine learning typically uses gradient compression to prevent gradient leakage attacks. This approach not only significantly reduces communication overhead but also maintains model performance. In this paper, we propose a method to reconstruct images from compressed gradients, called Deep Leakage from Compressed Gradients (DLCG). Extensive experiments on LeNet and ResNet20-4 demonstrate that our proposed method is able to reconstruct recognizable images from compressed gradients with sparsity levels as high as 90% and 80%, respectively, outperforming other methods. Furthermore, we analyze the sensitivity of gradient leakage attack to gradients of from different layers and propose a corresponding defense strategy.

Sparse Graph Representation Learning Based on Reinforcement Learning for Personalized Mild Cognitive Impairment (MCI) Diagnosis.

Resting-state functional magnetic resonance imaging (rs-fMRI) has gained attention as a reliable technique for investigating the intrinsic function patterns of the brain. It facilitates the extraction of functional connectivity networks (FCNs) that capture synchronized activity patterns among regions of interest (ROIs). Analyzing FCNs enables the identification of distinctive connectivity patterns associated with mild cognitive impairment (MCI). For MCI diagnosis, various sparse representation techniques have been introduced, including statistical- and deep learning-based methods. However, these methods face limitations due to their reliance on supervised learning schemes, which restrict the exploration necessary for probing novel solutions. To overcome such limitation, prior work has incorporated reinforcement learning (RL) to dynamically select ROIs, but effective exploration remains challenging due to the vast search space during training. To tackle this issue, in this study, we propose an advanced RL-based framework that utilizes a divide-and-conquer approach to decompose the FCN construction task into smaller sub-problems in a subject-specific manner, enabling efficient exploration under each sub-problem condition. Additionally, we leverage the learned value function to determine the sparsity level of FCNs, considering individual characteristics of FCNs. We validate the effectiveness of our proposed framework by demonstrating its superior performance in MCI diagnosis on publicly available cohort datasets.

Non-signal components minimization for sparse signal recovery

Sparse signal recovery is a challenging task. This paper focuses on the technique of non-signal components minimization for recovering the latent sparsest signal behind observations from two viewpoints: minimizing the residual subject to the sparsity level and minimizing the sparsity subject to the residual constraint. A successive sparsity increment method is proposed to address these two closely related tasks. This method yields two algorithms: SINSM for residual minimization and SISP for sparsity minimization. Additionally, an improved accelerated proximal gradient algorithm is provided to solve each non-signal components minimization problem penalized by the residual. The convergence of the improved algorithm is guaranteed. Numerical experiments demonstrate the superior performance of SINSM and SISP on both synthetic and real-world data.

Reconstructing Cancellous Bone From Down-Sampled Optical-Resolution Photoacoustic Microscopy Images With Deep Learning

ObjectiveBone diseases deteriorate the microstructure of bone tissue. Optical-resolution photoacoustic microscopy (OR-PAM) enables high spatial resolution of imaging bone tissues. However, the spatiotemporal trade-off limits the application of OR-PAM. The purpose of this study was to improve the quality of OR-PAM images without sacrificing temporal resolution. MethodsIn this study, we proposed the Photoacoustic Dense Attention U-Net (PADA U-Net) model, which was used for reconstructing full-scanning images from under-sampled images. Thereby, this approach breaks the trade-off between imaging speed and spatial resolution. ResultsThe proposed method was validated on resolution test targets and bovine cancellous bone samples to demonstrate the capability of PADA U-Net in recovering full-scanning images from under-sampled OR-PAM images. With a down-sampling ratio of [4, 1], compared to bilinear interpolation, the Peak Signal-to-Noise Ratio and Structural Similarity Index Measure values (averaged over the test set of bovine cancellous bone) of the PADA U-Net were improved by 2.325 dB and 0.117, respectively. ConclusionThe results demonstrate that the PADA U-Net model reconstructed the OR-PAM images well with different levels of sparsity. Our proposed method can further facilitate early diagnosis and treatment of bone diseases using OR-PAM.

A Sparsity-Invariant Model via Unifying Depth Prediction and Completion

The development of a sparse-invariant depth completion model capable of handling varying levels of input depth sparsity is highly desirable in real-world applications. However, existing sparse-invariant models tend to degrade when the input depth points are extremely sparse. In this paper, we propose a new model that combines the advantageous designs of depth completion and monocular depth estimation tasks to achieve sparse invariance. Specifically, we construct a dual-branch architecture with one branch dedicated to depth prediction and the other to depth completion. Additionally, we integrate the multi-scale local planar module in the decoders of both branches. Experimental results on the NYU Depth V2 benchmark and the OPPO prototype dataset equipped with the Spot-iToF316 sensor demonstrate that our model achieves reliable results even in cases with irregularly distributed, limited or absent depth information.

Clustering statistical method of high dimensional sparse data based on fuzzy data

Abstract Developing effective clustering and statistical methods for high-dimensional sparse data presents unique challenges compared to traditional low-dimensional data. To address this, a novel approach is proposed, leveraging fuzzy data principles to enhance the clustering and statistical performance of high-dimensional sparse datasets. The method builds upon the fuzzy C-means clustering algorithm, introducing key modifications for better suitability to high-dimensional sparse data. One crucial enhancement involves tackling the local optimization problem by optimizing the initial clustering center, significantly reducing clustering statistical time. Replacing the original Euclidean distance with cosine distance improves the clustering and statistical performance of high-dimensional sparse data. Experimental results have shown that this method has superior clustering statistical performance when the data dimensions are different. When the data dimension is low, and the blocking ratio is 10%, the clustering statistical effect is optimal. When the data dimension is high, and the blocking ratio is 40%, the clustering statistical effect is optimal. This method has higher hit rates and clustering statistical efficiency at different sparsity levels.

High-dimensional causal mediation analysis by partial sum statistic and sample splitting strategy in imaging genetics application.

Causal mediation analysis provides a systematic approach to explore the causal role of one or more mediators in the association between exposure and outcome. In omics or imaging data analysis, mediators are often high-dimensional, which brings new statistical challenges. Existing methods either violate causal assumptions or fail in interpretable variable selection. Additionally, mediators are often highly correlated, presenting difficulties in selecting and prioritizing top mediators. To address these issues, we develop a framework using Partial Sum Statistic and Sample Splitting Strategy, namely PS5, for high-dimensional causal mediation analysis. The method provides a powerful global mediation test satisfying causal assumptions, followed by an algorithm to select and prioritize active mediators with quantification of individual mediation contributions. We demonstrate its accurate type I error control, superior statistical power, reduced bias in mediation effect estimation, and accurate mediator selection using extensive simulations of varying levels of effect size, signal sparsity, and mediator correlations. Finally, we apply PS5 to an imaging genetics dataset of chronic obstructive pulmonary disease (COPD) patients ( N =8,897) in the COPDGene study to examine the causal mediation role of lung images ( p =5,810) in the associations between polygenic risk score and lung function and between smoking exposure and lung function, respectively. Both causal mediation analyses successfully estimate the global indirect effect and detect mediating image regions. Collectively, we find a region in the lower lobe of the right lung with a strong and concordant mediation effect for both genetic and environmental exposures. This suggests that targeted treatment toward this region might mitigate the severity of COPD due to genetic and smoking effects.

Physics-Informed Particle-Based Reinforcement Learning for Autonomy in Signalized Intersections

In this paper, we develop a framework to enhance the control of autonomous vehicles within signalized intersections by integrating system dynamics with imperfect sensor data. Although sensor data for autonomous vehicles is often partial, noisy, or missing, its alignment with underlying physics has the potential to significantly improve prediction accuracy. Our methodology involves a physics-based representation of system dynamics, accounting for stochastic elements originating from differences between real-world scenarios and model-based representations, using a Markov decision process (MDP) model that embraces system physics while accommodating uncertainties. Leveraging partial sensor data, we aim to diminish uncertainty associated with segments of the system model reflected in the data and better estimate the other physical variables that are not measurable through data. This framework develops particle-based reinforcement learning action policies, seamlessly integrating data and physics for controlling autonomous vehicles approaching signalized intersections. Numerical experiments showcase the effectiveness of these policies in ensuring safe control and decision-making in different scenarios of missing state variables and varying levels of data sparsity.

Behaviors of first-order optimizers in the context of sparse data and sparse models: A comparative study

The optimization algorithm plays a decisive role in the performance of deep learning models. Nevertheless, the task, the data properties, and the model's architecture directly influence this latter's effectiveness. Hence, analyzing the effect of different optimizers on specific tasks, such as change detection, is very advantageous due to the ambiguity surrounding the method selection. This paper is twofold. First, we investigate the performance of five first-order optimization methods, namely Momentum GD, NAG, AdaGrad, RMSProp, and Adam, in the context of change detection using a U-Net and DenseU-Net architectures for remote sensing images. To the best of our knowledge, no comparative study involving optimization methods has been conducted for change detection. The results reveal that the Adam optimizer is the preferred choice to train a U-Net like architecture for this specific task. Challenges like dataset size and irregular change patterns warrant further exploration and optimization. The second part of this study focuses on a detailed analysis of the Adam optimizer. We specifically examine its sensitivity to both model and data sparsity. To do this, we train several Convolutional Neural Network (CNN) model architectures, each with a different degree of sparsity, on several datasets with various levels of sparsity. The experimental results provide valuable insights, showing that the Adam optimizer is more influenced by model sparsity than data sparsity, and suggesting its suitability for effectively training densely connected models.