Sparse Structure Research Articles

DeMM: A Decoupled Matrix Multiplication Engine Supporting Relaxed Structured Sparsity

DeMM: A Decoupled Matrix Multiplication Engine Supporting Relaxed Structured Sparsity

MG-TCCA: Tensor Canonical Correlation Analysis across Multiple Groups.

Tensor Canonical Correlation Analysis (TCCA) is a commonly employed statistical method utilized to examine linear associations between two sets of tensor datasets. However, the existing TCCA models fail to adequately address the heterogeneity present in real-world tensor data, such as brain imaging data collected from diverse groups characterized by factors like sex and race. Consequently, these models may yield biased outcomes. In order to surmount this constraint, we propose a novel approach called Multi-Group TCCA (MG-TCCA), which enables the joint analysis of multiple subgroups. By incorporating a dual sparsity structure and a block coordinate ascent algorithm, our MG-TCCA method effectively addresses heterogeneity and leverages information across different groups to identify consistent signals. This novel approach facilitates the quantification of shared and individual structures, reduces data dimensionality, and enables visual exploration. To empirically validate our approach, we conduct a study focused on investigating correlations between two brain positron emission tomography (PET) modalities (AV-45 and FDG) within an Alzheimer's disease (AD) cohort. Our results demonstrate that MG-TCCA surpasses traditional TCCA and Sparse TCCA (STCCA) in identifying sex-specific cross-modality imaging correlations. This heightened performance of MG-TCCA provides valuable insights for the characterization of multimodal imaging biomarkers in AD.

Contrastive Multiview Attribute Graph Clustering With Adaptive Encoders.

Multiview attribute graph clustering aims to cluster nodes into disjoint categories by taking advantage of the multiview topological structures and the node attribute values. However, the existing works fail to explicitly discover the inherent relationships in multiview topological graph matrices while considering different properties between the graphs. Besides, they cannot well handle the sparse structure of some graphs in the learning procedure of graph embeddings. Therefore, in this article, we propose a novel contrastive multiview attribute graph clustering (CMAGC) with adaptive encoders method. Within this framework, the adaptive encoders concerning different properties of distinct topological graphs are chosen to integrate multiview attribute graph information by checking whether there exists high-order neighbor information or not. Meanwhile, the number of layers of the GCN encoders is selected according to the prior knowledge related to the characteristics of different topological graphs. In particular, the feature-level and cluster-level contrastive learning are conducted on the multiview soft assignment representations, where the union of the first-order neighbors from the corresponding graph pairs is regarded as the positive pairs for data augmentation and the sparse neighbor information problem in some graphs can be well dealt with. To the best of our knowledge, it is the first time to explicitly deal with the inherent relationships from the interview and intraview perspectives. Extensive experiments are conducted on several datasets to verify the superiority of the proposed CMAGC method compared with the state-of-the-art methods.

Graphformer: Adaptive graph correlation transformer for multivariate long sequence time series forecasting

Accurate long sequence time series forecasting (LSTF) remains a key challenge due to its complex time-dependent nature. Multivariate time series forecasting methods inherently assume that variables are interrelated and that the future state of each variable depends not only on its history but also on other variables. However, most existing methods, such as Transformer, cannot effectively exploit the potential spatial correlation between variables. To cope with the above problems, we propose a Transformer-based LSTF model, called Graphformer, which can efficiently learn complex temporal patterns and dependencies between multiple variables. First, in the encoder’s self-attentive downsampling layer, Graphformer replaces the standard convolutional layer with an dilated convolutional layer to efficiently capture long-term dependencies between time series at different granularity levels. Meanwhile, Graphformer replaces the self-attention mechanism with a graph self-attention mechanism that can automatically infer the implicit sparse graph structure from the data, showing better generality for time series without explicit graph structure and learning implicit spatial dependencies between sequences. In addition, Graphformer uses a temporal inertia module to enhance the sensitivity of future time steps to recent inputs, and a multi-scale feature fusion operation to extract temporal correlations at different granularity levels by slicing and fusing feature maps to improve model accuracy and efficiency. Our proposed Graphformer can improve the long sequence time series forecasting accuracy significantly when compared with that of SOTA Transformer-based models.

Factor Overnight GARCH-Itô Models

Abstract This article introduces a unified factor overnight GARCH-Itô model for large volatility matrix estimation and prediction. To account for whole-day market dynamics, the proposed model has two different instantaneous factor volatility processes for the open-to-close and close-to-open periods, while each embeds the discrete-time multivariate GARCH model structure. To estimate latent factor volatility, we assume the low rank plus sparse structure and employ nonparametric estimation procedures. Then, based on the connection between the discrete-time model structure and the continuous-time diffusion process, we propose a weighted least squares estimation procedure with the non-parametric factor volatility estimator and establish its asymptotic theorems.

On the Oracle Properties of Bayesian Random Forest for Sparse High-Dimensional Gaussian Regression

Random forest (RF) is a widely used data prediction and variable selection technique. However, the variable selection aspect of RF can become unreliable when there are more irrelevant variables than relevant ones. In response, we introduced the Bayesian random forest (BRF) method, specifically designed for high-dimensional datasets with a sparse covariate structure. Our research demonstrates that BRF possesses the oracle property, which means it achieves strong selection consistency without compromising the efficiency or bias.

Variance-Guided Structured Sparsity in Deep Neural Networks

The success of deep neural networks, especially convolutional neural networks in various applications, has greatly been possible by the presence of an enormous number of learnable parameters. These parameters increase the learning capacity of the model, but at the same time, it also significantly increases the computational and memory costs. This severely hinders the scalability of these models to limited resource environments, like IoT devices. The majority of the network weights are known to be redundant and can be removed from the network. This paper introduces a regularization scheme, which is the combination of structured sparsity regularization and variance regularization. It simultaneously helps to obtain computationally sparse models by making the majority of parameter groups zero and increasing the variance of non-zero groups to compensate for the accuracy drop. We use sparse group lasso, group sparsity variant of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\ell _{1}$</tex-math></inline-formula> (lasso) regularization, which removes redundant connections and unnecessary neurons from the network. For variance regularization, the KL divergence of the current parameter distribution and the target distribution is minimized, which aims to have the concentration of weights towards zero and a high variance of non-zero weights (skewed distribution). To check the effectiveness of the proposed regularizer, the experiments are performed on various benchmark datasets and it is noticed that variance regularization helps to reduce the accuracy drop caused by sparsity regularization. On MNIST, the trainable parameters are reduced from 331984 (baseline model) to 57327 and managed to obtain better accuracy than the baseline (99.6%). Also, on Fashion-MNIST, Cifar-10 and ImageNet the proposed scheme achieved state-of-the-art sparsity with almost no drop in accuracy.

Knowledge Transfer-Based Sparse Deep Belief Network.

Deep learning has made remarkable achievements in various applications in recent years. With the increasing computing power and the "black box" problem of neural networks, however, the development of deep neural networks (DNNs) has entered a bottleneck period. This article proposes a novel deep belief network (DBN) based on knowledge transfer and optimization of the network structure. First, a neural-symbolic model is proposed to extract rules to describe the dynamic operation mechanism of the deep network. Second, knowledge fusion is proposed based on the merge and deletion of the extracted rules from the DBN model. Finally, a new DNN, knowledge transfer-based sparse DBN (KT-SDBN) is constructed to generate a sparse network without excessive information loss. In comparison with DBN, KT-SDBN has a more sparse network structure and better learning performance on the existing knowledge and data. The experimental results in the benchmark data indicate that KT-SDBN not only has effective feature learning performance with 30% of the original network parameters but also shows a large compression rate that is far larger than other structure optimization algorithms.

Multiband Delay Estimation for Localization Using a Two-Stage Global Estimation Scheme

The time of arrival (TOA)-based localization techniques, which need to estimate the delay of the line-of-sight (LoS) path, have been widely employed in location-aware networks. To achieve a high-accuracy delay estimation, a number of multiband-based algorithms have been proposed recently, which exploit the channel state information (CSI) measurements over multiple non-contiguous frequency bands. However, to the best of our knowledge, there still lacks an efficient scheme that fully exploits the multiband gains when the phase distortion factors caused by hardware imperfections are considered, due to that the associated multi-parameter estimation problem contains many local optimums and the existing algorithms can easily get stuck in a “bad” local optimum. To address these issues, we propose a novel two-stage global estimation (TSGE) scheme for multiband delay estimation. In the coarse stage, we exploit the group sparsity structure of the multiband channel and propose a Turbo Bayesian inference (Turbo-BI) algorithm to achieve a good initial delay estimation based on a coarse signal model, which is transformed from the original multiband signal model by absorbing the carrier frequency terms. The estimation problem derived from the coarse signal model contains fewer local optimums and thus a more stable estimation can be achieved than directly using the original signal model. Then in the refined stage, with the help of coarse estimation results to narrow down the search range, we perform a global delay estimation using a particle swarm optimization-least square (PSO-LS) algorithm based on a refined multiband signal model to exploit the multiband gains to further improve the estimation accuracy. Simulation results show that the proposed TSGE significantly outperforms the benchmarks with comparative computational complexity.

Multi-material and thickness optimization of laminated composite structures subject to high-cycle fatigue

Laminated composite structures have a distinct inherent potential for optimization due to their tailorability and their associated complex failure mechanisms that makes intuitive design remarkably difficult. Optimization of such is a maturing technology with many criteria and manufacturing constraints having been successfully demonstrated. An approach for high-cycle fatigue is however yet to be developed in a gradient-based context. Thus, the objective of this work is to introduce a novel framework that allows for effective high-cycle fatigue optimization of laminated composite structures.Offset is taken in the Discrete Material and Thickness Optimization parametrization, which allows for simultaneous material and thickness selection for each layer that constitute a laminate. The fatigue analysis approach is based on accumulating damage from all variable-amplitude cycles in an arbitrary spectrum. As high-cycle fatigue behavior is highly nonlinear, it is difficult to handle in optimization. To stabilize the problem, damage is scaled using an inverse P-mean norm formulation that reduces the nonlinearity and provides an accurate measure of the damage. These scaled damages are then aggregated using P-norm functions to reduce the number of constraints. This is convenient, as it allows sensitivities to be efficiently calculated using analytical adjoint design sensitivity analysis. The effectiveness of this approach will be demonstrated on both benchmark examples and a more complicated main spar structure.

Ten-Fold Solvent Kinetic Isotope Effect for the Nonradiative Relaxation of the Aqueous Ferrate(VI) Ion.

Hypervalent iron intermediates have been invoked in the catalytic cycles of many metalloproteins, and thus, it is crucial to understand how the coupling between such species and their environment can impact their chemical and physical properties in such contexts. In this work, we take advantage of the solvent kinetic isotope effect (SKIE) to gain insight into the nonradiative deactivation of electronic excited states of the aqueous ferrate(VI) ion. We observe an exceptionally large SKIE of 9.7 for the nanosecond-scale relaxation of the lowest energy triplet ligand field state to the ground state. Proton inventory studies demonstrate that a single solvent O-H bond is coupled to the ion during deactivation, likely due to the sparse vibrational structure of ferrate(VI). Such a mechanism is consistent with that reported for the deactivation of f-f excited states of aqueous trivalent lanthanides, which exhibit comparably large SKIE values. This phenomenon is ascribed entirely to dissipation of energy into a higher overtone of a solvent acceptor mode, as any impact on the apparent relaxation rate due to a change in solvent viscosity is negligible.

On scaling robust feedback control and state estimation problems in power networks

Many mainstream robust control/estimation algorithms for power networks are designed using the Lyapunov theory as it provides performance guarantees for linear/nonlinear models of uncertain power networks but comes at the expense of scalability and sensitivity. In particular, Lyapunov-based approaches rely on forming semi-definite programs (SDPs) that are (i) not scalable and (ii) extremely sensitive to the choice of the bounding scalar that ensures the strict feasibility of the linear matrix inequalities (LMIs). This paper addresses these two issues by employing a celebrated non-Lyapunov approach (NLA) from the control theory literature. In lieu of linearized models of power grids, we focus on (the more representative) nonlinear differential algebraic equation (DAE) models and showcase the simplicity, scalability, and parameter-resiliency of NLA. For some power systems, the approach is nearly fifty times faster than solving SDPs via standard solvers with almost no impact on the performance. The case studies also demonstrate that NLA can be applied to more realistic scenarios in which (i) only partial state data is available and (ii) sparsity structures are imposed on the feedback gain. The paper also showcases that virtually no degradation in state estimation quality is experienced when applying NLA.

SPADA: A Sparse Approximate Data Structure Representation for Data Plane Per-flow Monitoring

Accurate per-flow monitoring is critical for precise network diagnosis, performance analysis, and network operation and management in general. However, the limited amount of memory available on modern programmable devices and the large number of active flows force practitioners to monitor only the most relevant flows with approximate data structures, limiting their view of network traffic. We argue that, due to the skewed nature of network traffic, such data structures are, in practice, heavily underutilized, i.e. sparse, thus wasting a significant amount of memory. This paper proposes a Sparse Approximate Data Structure (SPADA) representation that leverages sparsity to reduce the memory footprint of per-flow monitoring systems in the data plane while preserving their original accuracy. SPADA representation can be integrated into a generic per-flow monitoring system and is suitable for several measurement use cases. We prototype SPADA in P4 for a commercial FPGA target and test our approach with a custom simulator that we make publicly available, on four real network traces over three different monitoring tasks. Our results show that SPADA achieves 2× to 11× memory footprint reduction with respect to the state-of-the-art while maintaining the same accuracy, or even improving it.

Effect of stitch density on the damage inhibition and compression strength after high-velocity impact of UHMWPE fiber composites

This paper investigates the effect of stitch density on the high-velocity impact damage inhibition and CAI (compression after impact) strength of UHMWPE (Ultra-High Molecular Weight Polyethylene) fiber composites. The damage response and compression behavior after impact were quantified and analyzed using CT scanning and DIC (Digital Image Correlation) techniques. The findings reveal that the delamination area of the composite plate with the stitch space of 1 cm reduces by 49.14 % compared to the unstitched sample, and exhibits a 73.05 % higher compressive load than the plate with a 4 cm stitch space. Notably, the 4 cm stitch composite plates demonstrate a significant advantage in energy absorption. However, the sparse stitch structure destroys the original structure inside the composite, resulting in a degradation of CAI performance. The study provides guidance for the practical design of composite materials with enhanced energy absorption and damage resistance.

Causal inference of general treatment effects using neural networks with a diverging number of confounders

Semiparametric efficient estimation of various multi-valued causal effects, including quantile treatment effects, is important in economic, biomedical, and other social sciences. Under the unconfoundedness condition, adjustment for confounders requires estimating the nuisance functions relating outcome or treatment to confounders nonparametrically. This paper considers a generalized optimization framework for efficient estimation of general treatment effects using artificial neural networks (ANNs) to approximate the unknown nuisance function of growing-dimensional confounders. We establish a new approximation error bound for the ANNs to the nuisance function belonging to a mixed smoothness class without a known sparsity structure. We show that the ANNs can alleviate the “curse of dimensionality” under this circumstance. We establish the root-n consistency and asymptotic normality of the proposed general treatment effects estimators, and apply a weighted bootstrap procedure for conducting inference. The proposed methods are illustrated via simulation studies and a real data application.

Innovative chaotic dragon fractal (ChDrFr) shapes for efficient encryption applications: a new highly secure image encryption algorithm

In this paper, the generation of new dragon fractal shapes with chaotic iteration parameters is introduced as the main component of a new efficient approach for different cryptographic applications. This process involves applying a chaotic map, which is considered the initiator pattern, to generate different chaotic dragon fractal (ChDrFr) shapes in lieu of lines (which are classically used to generate dragon fractals). This is the new concept of this paper. The used chaotic maps are sensitive to their initial conditions and are characterized by randomness; hence, the resulting scheme is highly secure. As the resulting ChDrFr shapes have sparse structures, the spaces are packed with random values generated from another 5D hyper chaotic map. For encryption applications based on the substitution approach, one of the five generated ChFrDr shapes can be used to construct a chaotic fractal (ChFr) S-Box, while the other four ChDrFr shapes can be used for diffusion purposes. As an application to these new ChDrFr shapes and the ChFr S-Box, we introduce in this paper a new highly secure image encryption algorithm. A Henon chaotic map is used as the initiator of the ChDrFr shapes. The integer wavelet transform (IWT) is used to generate an approximation and three detail sub-bands for the original image. As the approximation sub-band contains a considerable amount of information about the original image, the above-described ChFr S-Box is used as a replacement for each pixel’s value in this sub-band. Then, the resultant substituted image is diffused with one of the generated ChFrDr shapes. The other three ChDrFr shapes are XORed with the details sub-images. Numerical simulation is applied to ensure the efficacy of encrypted images against different attacks. In particular, the correlation coefficient between the initial and the generated images is shown to be nearly zero. Moreover, tests reveal that the information entropy of the encrypted images and UACI were close to their optimum values. The properties of the newly proposed ChDrFr-based encryption algorithm are compared to the ones obtained by other encryption algorithms, and the results prove the superiority of this newly proposed algorithm to other types of encryption methods.

Innovative and entrepreneurial characteristics of university students based on logistic regression model

Abstract This paper proposes a logistic regression model with structural sparsity to study the characteristics of innovation and entrepreneurship among college students. The article first analyzes the basic form of the logistic regression model, including the objective function and the selection method for the penalty function. Then, because the ADMM algorithm combines the advantages of augmented Lagrangian and pairwise decomposition, which can reduce the computational difficulty and complexity, based on this advantage, this paper designs the ADMM algorithm solution framework that is favorable for distributed computing. Finally, this paper analyzes the relationship between the development of innovation and entrepreneurship ability of students in R colleges and universities and their gender, grade, academic foundation, experience in clubs and discipline type. The results yielded that college students’ mean value of innovation and entrepreneurship competence in HEI R was 3.734. The mean value of the scores of each sub-competence ranged from 3.531 to 3.918, which puts the overall innovation and entrepreneurship competence of students in this university at an intermediate level. Therefore, this study plays an important role in understanding the innovation and entrepreneurial characteristics of students in higher education.

Research on the safety and practicability of the combination of offshore wind and solar energy

Wind power is a renewable way of power generation. It features simple scale construction, considerable efficiency and clean and pollution-free. The current situation of offshore wind power generation is also facing many challenges, safety, economy, feasibility. In this paper, the design method is adopted and the structure is checked and analyzed, and the current problems and solutions of offshore wind power generation are analyzed from the perspectives of structural safety, cost estimation and design feasibility. Through practical design and calculation, this study makes a breakthrough in the structure of offshore floating wind power generation, and provides a solution for the spar structure to limit the sea state.

Scalable genetic screening for regulatory circuits using compressed Perturb-seq

Pooled CRISPR screens with single-cell RNA sequencing readout (Perturb-seq) have emerged as a key technique in functional genomics, but they are limited in scale by cost and combinatorial complexity. In this study, we modified the design of Perturb-seq by incorporating algorithms applied to random, low-dimensional observations. Compressed Perturb-seq measures multiple random perturbations per cell or multiple cells per droplet and computationally decompresses these measurements by leveraging the sparse structure of regulatory circuits. Applied to 598 genes in the immune response to bacterial lipopolysaccharide, compressed Perturb-seq achieves the same accuracy as conventional Perturb-seq with an order of magnitude cost reduction and greater power to learn genetic interactions. We identified known and novel regulators of immune responses and uncovered evolutionarily constrained genes with downstream targets enriched for immune disease heritability, including many missed by existing genome-wide association studies. Our framework enables new scales of interrogation for a foundational method in functional genomics.

Nonparametric quantile scalar-on-image regression

A quantile scalar-on-image regression model is developed to comprehensively study the relationship between cognitive decline and various clinical covariates and imaging factors. As a motivating example, the high-dimensional brain imaging data from the research on Alzheimer's disease are considered predictors of patients' cognitive decline. A Bayesian nonparametric model is proposed to handle the complex spatially distributed imaging data, where the coefficient function is assumed to be a latent Gaussian process. A soft-thresholding operator is introduced to capture the sparse structure of the regression coefficients. Utilizing kernel basis functions to approximate the latent Gaussian process facilitates easy-to-implement computation and consistent estimation. Inference is performed within the Bayesian framework, using an efficient Markov chain Monte Carlo algorithm. The proposed method is compared with the functional principal component analysis method in simulations and applied to a study of Alzheimer's disease.