Sparsity Level Research Articles

BiGCAN: A novel SRS-based bidirectional graph Convolution Attention Network for dynamic user preference and next-item recommendation

Bidirectional graph modeling represents a significant emerging technology that can be effectively integrated into sequential recommender systems. This innovation enables the explicit modeling of both forward and backward interactions in dynamic user preferences and next-item recommendations. However, existing approaches, particularly unidirectional methods, suffer from critical limitations. Unidirectional models are restricted to processing information in a single direction, which limits their ability to fully capture user interaction patterns, leading to suboptimal recommendation performance. Additionally, simplistic fusion mechanisms in current methods fail to effectively learn the intricate relationships between long-term and short-term user preferences. To address these limitations, we propose a novelty of the Bidirectional Graph Convolution Attention Network (BiGCAN) to enhance recommendation quality through three key innovations. First, bidirectional graph convolution network captures long-term preferences by modeling interactions in both forward and backward directions, allowing for a more comprehensive understanding of user behavior over time. Second, time-aware bidirectional graph attention network focuses on short-term preferences by incorporating temporal dynamics, enabling the model to adapt to recent changes in user preferences. Third, Bidirectional Gated Collaborative Filtering (BiGatedCF) mechanism integrates both long-term and short-term preferences through an advanced gating mechanism that efficiently balances these interactions. The BiGatedCF module combines a bidirectional gated recurrent unit to capture complex sequential dependencies, a deep matrix factorization to learn high-quality latent representations, and a multi-layer perceptron to match scores. Empirical experiments on four benchmark datasets demonstrate that BiGCAN surpasses state-of-the-art methods in terms of Top-N recommendations and performance across varying sparsity levels.

Optimal sequential detection by sparsity likelihood

We propose here a sparsity likelihood stopping rule to detect change-points when there are multiple data streams. It is optimal in the sense of minimizing, asymptotically, the detection delay when the change-points is present in only a small fraction of the data streams. This optimality holds at all levels of change-point sparsity. A key contribution of this paper is that we show optimality when there is extreme sparsity. Extreme sparsity refers to the number of data streams with change-points increasing very slowly as the number of data streams goes to infinity. The theoretical results are backed by a numerical study that shows the sparsity likelihood stopping rule performing well at all levels of sparsity. Applications of the stopping rule on non-normal models are also illustrated here.

Technical note: Optimizing spot-scanning proton arc therapy with a novel spot sparsity approach.

One of the main challenges of utilizing spot-scanning proton arc therapy (SPArc) in routine clinics is treatment delivery efficiency. Spot reduction, which relies on spot sparsity optimization (SSO), is crucial for achieving high delivery efficiency inSPArc. This study aims to develop a novel SSO approach based on the alternating directions method of multipliers (ADMM) for SPArc to achieve high treatment delivery efficiency and maintain optimal dosimetric planquality. In this study, SSO for SPArc is based on the least-square dose fidelity term with L0-norm regularization. The novel optimization approach is based on the ADMM framework, in which the minimum monitor unit constraint was considered to improve the plan quality. A state-of-the-art SSO method, the primal-dual active set with continuation (PDASC) algorithm published previously, was utilized as a benchmark. Two SPArc plan groups with the same beam assignment and clinical constraint were generated, in which the former group was SPArc plan with SSO utilizing ADMM, denoted as , and the later group was SPArc with SSO utilizing PDASC, denoted as . Nine clinical cases included five different cancer sites (brain, lung, liver, prostate, and head&neck cancer) were used. The SSO method's performance was evaluated in terms of spot sparsity level (the number of zero-valued elements divided by the total number of elements), beam delivery time, dosimetric plan quality, and plan robustness. Compared to the plan, the plan exhibits superior sparsity and higher delivery efficiency while maintaining good planquality. This study introduces a novel spot sparsity optimization approach using the ADMM framework to improve the delivery efficiency of SPArc. Compared to the existing state-of-the-art SSO method, such an approach could further enhance delivery efficiency while maintaining good plan quality, which could promote the implementation of SPArc in theclinic's.

Online Learning and Decision Making Under Generalized Linear Model with High-Dimensional Data

We propose a minimax concave penalized multiarmed bandit algorithm under the generalized linear model (G-MCP-Bandit) for decision-makers facing high-dimensional data in an online learning and decision-making environment. We demonstrate that in the data-rich regime, the G-MCP-Bandit algorithm attains the optimal cumulative regret in the sample size dimension and a tight bound in the covariate dimension and the significant covariate dimension. In the data-poor regime, the G-MCP-Bandit algorithm maintains a tight regret upper bound. In addition, we develop a local linear approximation method, the two-step weighted Lasso procedure, to identify the minimax concave penalty (MCP) estimator for the G-MCP-Bandit algorithm when samples are not independent and identically distributed. Under this procedure, the MCP estimator can match the oracle estimator with high probability and converge to the true parameters at the optimal convergence rate. Finally, through experiments based on both synthetic and real data sets, we show that the G-MCP-Bandit algorithm outperforms other benchmarking algorithms in terms of cumulative regret and that the benefits of the G-MCP-Bandit algorithm increase in the data’s sparsity level and the size of the decision set. This paper was accepted by J. George Shanthikumar, big data analytics. Funding: This work was supported by the National Natural Science Foundation of China (NSFC) [Grants W2441021, 72371172, 72342023, and 71929101] and the National Science Foundation (NSF) [Grant DMS 1820702]. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2022.01557 .

Adaptive decomposition-based evolutionary algorithm for many-objective optimization with two-stage dual-density judgment

In order to better balance the convergence and diversity of MOEA/D for many objective optimization problems (MaOPs) with various Pareto fronts (PFs), an adaptive decomposition-based evolutionary algorithm for MaOPs with two-stage dual-density judgment is proposed. To solve the problem that weighted Tchebycheff decomposition may produce weakly Pareto optimal solutions when the solution is not unique or the uniqueness is difficult to guarantee, an augmented weighted Tchebycheff decomposition is adopted. To balance the convergence and diversity of non-dominated solutions in the external archive, different sparsity-level evaluations using vector angles or Euclidean distances are used to measure the distribution of solutions at different stages. To improve the diversity of solution sets obtained by MOEA/D for various PFs, an adaptive weight vector adjustment method based on two-stage dual-density judgment is presented. For weight vector addition, the potential search area is found according to the two-stage density judgment, and then a two-stage sparsity level judgment on the solutions of this area is performed for a second density judgment. For weight vector deletion, the degree of crowding is used to delete the weight vectors with a high crowding degree. Compared with nine advanced multi-objective optimization algorithms on DTLZ and WFG problems, the results demonstrate that the performance of the proposed algorithm is significantly better than other algorithms.

DSGI‐Net: Density‐based Selective Grouping Point Cloud Learning Network for Indoor Scene

AbstractIndoor scene point clouds exhibit diverse distributions and varying levels of sparsity, characterized by more intricate geometry and occlusion compared to outdoor scenes or individual objects. Despite recent advancements in 3D point cloud analysis introducing various network architectures, there remains a lack of frameworks tailored to the unique attributes of indoor scenarios. To address this, we propose DSGI‐Net, a novel indoor scene point cloud learning network that can be integrated into existing models. The key innovation of this work is selectively grouping more informative neighbor points in sparse regions and promoting semantic consistency of the local area where different instances are in proximity but belong to distinct categories. Furthermore, our method encodes both semantic and spatial relationships between points in local regions to reduce the loss of local geometric details. Extensive experiments on the ScanNetv2, SUN RGB‐D, and S3DIS indoor scene benchmarks demonstrate that our method is straightforward yet effective.

Variational Bayesian EM Algorithm for Quantile Regression in Linear Mixed Effects Models

This paper extends the normal-beta prime (NBP) prior to Bayesian quantile regression in linear mixed effects models and conducts Bayesian variable selection for the fixed effects of the model. The choice of hyperparameters in the NBP prior is crucial, and we employed the Variational Bayesian Expectation–Maximization (VBEM) for model estimation and variable selection. The Gibbs sampling algorithm is a commonly used Bayesian method, and it can also be combined with the EM algorithm, denoted as GBEM. The results from our simulation and real data analysis demonstrate that both the VBEM and GBEM algorithms provide robust estimates for the hyperparameters in the NBP prior, reflecting the sparsity level of the true model. The VBEM and GBEM algorithms exhibit comparable accuracy and can effectively select important explanatory variables. The VBEM algorithm stands out in terms of computational efficiency, significantly reducing the time and resource consumption in the Bayesian analysis of high-dimensional, longitudinal data.

Analyses of the tail-[formula omitted] minimization for fast and enhanced sparse selections

We investigate the effectiveness and efficiency of the iterative tail-ℓ2 minimization (tail-ℓ2-min) technique for its sparse selection capabilities. We conduct profile analyses on the tail-ℓ2-min, establishing the equivalence of the tail-ℓ2-min problem to a two-stage profile ℓ2 formulation, both featuring analytical solutions. The tail null space property (NSP) of sensing matrix A is shown to be equivalent to the NSP of the newly defined profile matrix Ã. Besides the error bound analysis for the tail-ℓ2-min under the typical tail-NSP condition, a novel error bound of the tail-ℓ2-min formulation is also established without relying on NSP or restricted isometry property (RIP) assumptions. It merely contains tractable coefficients of A, and offers insights into successful recovery, with the observation of the convergent iterative procedure. Numerical studies and the applications to image reconstruction demonstrate the superiority and fast convergence of the tail-ℓ2 sparse solution over state-of-the-art sparse selection methodologies. The sparsity level of a signal that the tail-ℓ2 profile algorithm guarantees the recovery is around 41% higher than that of the basis pursuit algorithm. The analytical solutions of the tail-ℓ2 method at each iteration also ensure that the tail-ℓ2 sparse recovery process is notably fast, especially for high dimensions and high sparsity levels.

Comparison of Reservoir Computing topologies using the Recurrent Kernel approach

Reservoir Computing (RC) has become popular in recent years thanks to its fast and efficient computational capabilities. Standard RC has been shown to be equivalent in the asymptotic limit to Recurrent Kernels, which helps in analyzing its expressive power. However, many well-established RC paradigms, such as Leaky RC, Sparse RC, and Deep RC, are yet to be systematically analyzed in such a way. We define the Recurrent Kernel limit of all these RC topologies and conduct a convergence study for a wide range of activation functions and hyperparameters. Our findings provide new insights into various aspects of Reservoir Computing. First, we demonstrate that there is an optimal sparsity level which grows with the reservoir size. Furthermore, our analysis suggests that Deep RC should use reservoir layers of decreasing sizes. Finally, we perform a benchmark demonstrating the efficiency of Structured Reservoir Computing compared to vanilla and Sparse Reservoir Computing.

Neural architecture search for adversarial robustness via learnable pruning

The convincing performances of deep neural networks (DNNs) can be degraded tremendously under malicious samples, known as adversarial examples. Besides, with the widespread edge platforms, it is essential to reduce the DNN model size for efficient deployment on resource-limited edge devices. To achieve both adversarial robustness and model sparsity, we propose a robustness-aware search framework, an Adversarial Neural Architecture Search by the Pruning policy (ANAS-P). The layer-wise width is searched automatically via the binary convolutional mask, titled Depth-wise Differentiable Binary Convolutional indicator (D2BC). By conducting comprehensive experiments on three classification data sets (CIFAR-10, CIFAR-100, and Tiny-ImageNet) utilizing two adversarial losses TRADES (TRadeoff-inspired Adversarial DEfense via Surrogate-loss minimization) and MART (Misclassification Aware adveRsarial Training), we empirically demonstrate the effectiveness of ANAS in terms of clean accuracy and adversarial robust accuracy across various sparsity levels. Our proposed approach, ANAS-P, outperforms previous representative methods, especially in high-sparsity settings, with significant improvements.

A Novel Topology Adaptation Strategy for Dynamic Sparse Training in Deep Reinforcement Learning.

Deep reinforcement learning (DRL) has been widely adopted in various applications, yet it faces practical limitations due to high storage and computational demands. Dynamic sparse training (DST) has recently emerged as a prominent approach to reduce these demands during training and inference phases, but existing DST methods achieve high sparsity levels by sacrificing policy performance as they rely on the absolute magnitude of connections for pruning and randomly generating connections. Addressing this, our study presents a generic method that can be seamlessly integrated into existing DST methods in DRL to enhance their policy performance while preserving their sparsity levels. Specifically, we develop a novel method for calculating the importance of connections within the model. Subsequently, we dynamically adjust the sparse network topology by dropping existing connections and introducing new connections based on their respective importance values. Through validation on eight widely used simulation tasks, our method improves two state-of-the-art (SOTA) DST approaches by up to 70% in episode return and average return across all episodes under various sparsity levels.

Leveraging discriminative data: A pathway to high-performance, stable One-shot Network Pruning at Initialization

One-shot Network Pruning at Initialization (OPaI) is acknowledged as a highly cost-effective strategy for network pruning. However, it has been observed that OPaI models tend to suffer from reduced accuracy stability as target sparsity increases. This study introduces a novel approach by incorporating Discriminative Data (DD) into OPaI, significantly improving performance at higher sparsity levels while maintaining the “one-shot” nature. Our approach achieves state-of-the-art (SOTA) performance, challenging the previously held belief of OPaI’s data independence. Through detailed ablation studies, we thoroughly investigate the influence of data on OPaI, particularly focusing on how DD addresses a common failure in OPaI known as “layer collapse”. Furthermore, our experiments demonstrate that leveraging DD from various pre-trained models can markedly boost pruning performance across different models without requiring changes to the existing model architectures or pruning methodologies. These significant improvements highlight our method’s high generalizability and stability, paving new paths for advancing pruning strategies. Our code is publicly available at: https://github.com/Nonac/DDOPaI.

Sqrt-Loglogish CNN and Markov model for 5G spectrum sensing application

The research presents innovative methods for spectrum sensing in 5G networks, using the Sqrt-Loglogish convolutional neural network (SL-CNN) and hidden orthogonal intuitionistic fuzzy Markov model (HOIFMM). The proposed methods aim to tackle issues related to detecting principal user signals accurately, mitigating interference, and efficiently utilizing the spectrum in wideband spectrum environments due to their diverse and ever changing characteristics. The Sqrt-Loglogish CNN improves spectrum sensing by addressing static threshold dependency and potential overfitting. The HOIFMM offers a complex framework for predicting sparsity levels and primary user patterns. The results highlight the effectiveness of the suggested techniques in differentiating primary user signals from noise and interference, resulting in enhanced interference management tactics and overall network performance. MATLAB simulation is performed and compared the proposed methods performance with existing state-of-the-art methods such as CNN, deep neural network (DNN), long short-term memory (LSTM) and artificial neural networks (ANN). The proposed method has outperformed existing methods in terms of sensitivity, accuracy, and precision. Future endeavors include improving these methods, investigating sophisticated machine learning algorithms, and doing real world validations to guarantee scalability and resilience in various 5G deployment situations. This research advances the spectrum sensing capabilities in 5G networks, potentially improving efficiency, reliability, and quality of service.

On learning sparse linear models from cross samples

The sample complexity of a sparse linear model where samples are dependent is studied in this paper. We consider a specific dependency structure of the samples which arises in some experimental designs such as drug sensitivity studies, where two sets of objects (drugs and cells) are sampled independently, and after crossing (making all possible combinations of drugs and cells), the resulting output (efficacy of drugs) is measured. We call these types of samples as “cross samples”. The dependency among such samples is strong, and existing theoretical studies are either inapplicable or fail to provide realistic bounds. We aim at analyzing the performance of the Lasso estimator where the underlying distributions are mixtures of Gaussians and the data dependency arises from the crossing procedure. Our theoretical results show that the performance of the Lasso estimator in case of cross samples follows that of the i.i.d. samples with differences in constant factors. Through numerical results, we observe a phase transition: When datasets are too small, the error for cross samples is much larger than for i.i.d. samples, but once the size is large enough, cross samples are nearly as useful as i.i.d. samples. Our theoretical analysis suggests that the transition threshold is governed by the level of sparsity of the true parameter vector being estimated.

Robust sparse recovery with sparse Bernoulli matrices via expanders

Sparse binary matrices are of great interest in the field of sparse recovery, nonnegative compressed sensing, statistics in networks, and theoretical computer science. This class of matrices makes it possible to perform signal recovery with lower storage costs and faster decoding algorithms. In particular, Bernoulli (p) matrices formed by independent identically distributed (i.i.d.) Bernoulli (p) random variables are of practical relevance in the context of noise-blind recovery in nonnegative compressed sensing.In this work, we investigate the robust nullspace property of Bernoulli (p) matrices. Previous results in the literature establish that such matrices can accurately recover n-dimensional s-sparse vectors with m=O(sc(p)log⁡ens) measurements, where c(p)≤p is a constant dependent only on the parameter p. These results suggest that in the sparse regime, as p approaches zero, the (sparse) Bernoulli (p) matrix requires significantly more measurements than the minimal necessary, as achieved by standard isotropic subgaussian designs. However, we show that this is not the case.Our main result characterizes, for a wide range of sparsity levels s, the smallest p for which sparse recovery can be achieved with the minimal number of measurements. We also provide matching lower bounds to establish the optimality of our results and explore connections with the theory of invertibility of discrete random matrices and integer compressed sensing.

Optimal Network Pairwise Comparison

We are interested in the problem of two-sample network hypothesis testing: given two networks with the same set of nodes, we wish to test whether the underlying Bernoulli probability matrices of the two networks are the same or not. We propose Interlacing Balance Measure (IBM) as a new two-sample testing approach. We consider the Degree-Corrected Mixed-Membership (DCMM) model for undirected networks, where we allow severe degree heterogeneity, mixed-memberships, flexible sparsity levels, and weak signals. In such a broad setting, how to find a test that has a tractable limiting null and optimal testing performances is a challenging problem. We show that IBM is such a test: in a broad DCMM setting with only mild regularity conditions, IBM has N ( 0 , 1 ) as the limiting null and achieves the optimal phase transition. While the above is for undirected networks, IBM is a unified approach and is directly implementable for directed networks. For a broad directed-DCMM (extension of DCMM for directed networks) setting, we show that IBM has N ( 0 , 1 / 2 ) as the limiting null and continues to achieve the optimal phase transition. We have also applied IBM to the Enron email network and a gene co-expression network, with interesting results.

Improved gradient leakage attack against compressed gradients in federated learning

Distributed machine learning, such as federated learning, protects privacy by collecting gradients instead of training data. Recent studies have shown that gradient leakage attacks are possible in distributed machine learning, that is, the training data can be reconstructed from the shared gradients. In practical applications, distributed machine learning typically uses gradient compression to prevent gradient leakage attacks. This approach not only significantly reduces communication overhead but also maintains model performance. In this paper, we propose a method to reconstruct images from compressed gradients, called Deep Leakage from Compressed Gradients (DLCG). Extensive experiments on LeNet and ResNet20-4 demonstrate that our proposed method is able to reconstruct recognizable images from compressed gradients with sparsity levels as high as 90% and 80%, respectively, outperforming other methods. Furthermore, we analyze the sensitivity of gradient leakage attack to gradients of from different layers and propose a corresponding defense strategy.

Sparse Graph Representation Learning Based on Reinforcement Learning for Personalized Mild Cognitive Impairment (MCI) Diagnosis.

Resting-state functional magnetic resonance imaging (rs-fMRI) has gained attention as a reliable technique for investigating the intrinsic function patterns of the brain. It facilitates the extraction of functional connectivity networks (FCNs) that capture synchronized activity patterns among regions of interest (ROIs). Analyzing FCNs enables the identification of distinctive connectivity patterns associated with mild cognitive impairment (MCI). For MCI diagnosis, various sparse representation techniques have been introduced, including statistical- and deep learning-based methods. However, these methods face limitations due to their reliance on supervised learning schemes, which restrict the exploration necessary for probing novel solutions. To overcome such limitation, prior work has incorporated reinforcement learning (RL) to dynamically select ROIs, but effective exploration remains challenging due to the vast search space during training. To tackle this issue, in this study, we propose an advanced RL-based framework that utilizes a divide-and-conquer approach to decompose the FCN construction task into smaller sub-problems in a subject-specific manner, enabling efficient exploration under each sub-problem condition. Additionally, we leverage the learned value function to determine the sparsity level of FCNs, considering individual characteristics of FCNs. We validate the effectiveness of our proposed framework by demonstrating its superior performance in MCI diagnosis on publicly available cohort datasets.

Non-signal components minimization for sparse signal recovery

Sparse signal recovery is a challenging task. This paper focuses on the technique of non-signal components minimization for recovering the latent sparsest signal behind observations from two viewpoints: minimizing the residual subject to the sparsity level and minimizing the sparsity subject to the residual constraint. A successive sparsity increment method is proposed to address these two closely related tasks. This method yields two algorithms: SINSM for residual minimization and SISP for sparsity minimization. Additionally, an improved accelerated proximal gradient algorithm is provided to solve each non-signal components minimization problem penalized by the residual. The convergence of the improved algorithm is guaranteed. Numerical experiments demonstrate the superior performance of SINSM and SISP on both synthetic and real-world data.

Reconstructing Cancellous Bone From Down-Sampled Optical-Resolution Photoacoustic Microscopy Images With Deep Learning

ObjectiveBone diseases deteriorate the microstructure of bone tissue. Optical-resolution photoacoustic microscopy (OR-PAM) enables high spatial resolution of imaging bone tissues. However, the spatiotemporal trade-off limits the application of OR-PAM. The purpose of this study was to improve the quality of OR-PAM images without sacrificing temporal resolution. MethodsIn this study, we proposed the Photoacoustic Dense Attention U-Net (PADA U-Net) model, which was used for reconstructing full-scanning images from under-sampled images. Thereby, this approach breaks the trade-off between imaging speed and spatial resolution. ResultsThe proposed method was validated on resolution test targets and bovine cancellous bone samples to demonstrate the capability of PADA U-Net in recovering full-scanning images from under-sampled OR-PAM images. With a down-sampling ratio of [4, 1], compared to bilinear interpolation, the Peak Signal-to-Noise Ratio and Structural Similarity Index Measure values (averaged over the test set of bovine cancellous bone) of the PADA U-Net were improved by 2.325 dB and 0.117, respectively. ConclusionThe results demonstrate that the PADA U-Net model reconstructed the OR-PAM images well with different levels of sparsity. Our proposed method can further facilitate early diagnosis and treatment of bone diseases using OR-PAM.