Sparse Weight Research Articles

Feature fusion and distillation embedded sparse Bayesian learning model for in-situ foreknowledge of robotic machining errors

In recent years, robotic machining of complex surface has become a research hotspot for intelligent manufacture. Considering the weak rigidity characteristics of industrial robot structure, the machining errors will be generated during the robotic machining process, which will further induce the undesirable impact on machining quality and production cycles. Therefore, this work proposes an in-situ foreknowledge model for robotic machining errors of complex curved parts. The static and dynamic features during the robotic manufacture process are obtained, where the robots’ stiffness concerning end effect operator is extracted as the static features, as well as the dynamic features of the monitored dynamic cutting forces. Furthermore, the structured fusion features are utilized as the dataset to train the foreknowledge model based on sparse Bayesian learning method. In order to adapt to the rapid response in the industrial field, the principal component analysis strategy is adopted to reduce feature dimension, by which the complexity of fusion features is reduced by 50.51 %. In addition, based on the adjustment of sparse weights, the complexity of the sparse Bayesian foreknowledge model is reduced by 74.31 %. According to the validation experiment on the blade shaped workpiece of marine propeller, the results indicate that the prediction accuracy of the proposed in-situ foreknowledge model is in the range of 60.03–87.07 µm, meeting the engineering requirements of robotic machining errors. The proposed in-situ foreknowledge sparse model displays the potential prospects for robotic machining errors prediction and control in the industrial application field.

Penalized Mallow’s model averaging

This article proposes penalized Mallow’s model averaging (pMMA) in the linear regression framework given non nested candidate models. Compared to the MMA, additional constraints are imposed on model weights. We introduce a general framework and allow for non convex constraints such as SCAD, MCP, and TLP. We establish the asymptotic optimality of our proposed penalized MMA (pMMA) estimator and show that the pMMA can achieve a higher sparsity level than the classic MMA. A coordinate-wise descent algorithm has been developed to compute the pMMA estimator efficiently. We conduct simulation and empirical studies to show that our pMMA estimator produces a more sparse weight vector than the MMA, but with better out-of-sample performance.

A spatial panel autoregressive model specification with inverse quantile separation distances of locations

This paper proposes an alternative spatial weight that efficiently captures a spatial dependence. In the past, researchers often used sparse or inverse distance spatial weights. A dense spatial weight is defined by partitioning the separation distances between locations based on quantile values over large spatial scales, where each partition forms conjoint neighborhood sectors and the weights of the respective inverse quantile of separation distances are assigned to each sector. Instead of joint modeling of the spatiotemporal process, a simultaneous spatial panel model is employed after the panel component is imposed on the proposed spatial weight using Kronecker product to perform maximum likelihood estimation and Bayesian inference via MCMC Gibbs method. The specification also involves space, time, and space–time simultaneous components. The performance of the models for the proposed spatial weight is compared with the existing spatial weights using parameter bias. A smaller value of the bias close to zero indicates a stronger value of the spatial error parameter for the proposed spatial weight over the existing spatial weights. It also induces spatial auto-correlations both in the spatial panel data of neighboring locations as well as in the errors. Thus, the proposed method affirms the best efficiency for the dynamic combined spatial panel autoregressive model with random effect specification over the lag and error models, and fixed effect specification.

Complex sparse Bayesian learning for guided wave dispersion curve estimation in plate-like structures

This paper proposes a new dispersion curve estimation method that employs the complex sparse Bayesian learning (CSBL). It is well accepted that guided wave packets are distorted because of the differences in propagation velocities at different frequencies; thus, the preceding velocity–frequency curve estimation is beneficial for wave packet recovery, feature recognition and defect localization. Conventional dispersion curve estimation methods, such as two-dimensional Fourier transform and multiple signal classification, are suitable for array signal and are restricted by the transducer aperture, leading to a small application scope. According to the frequency–response model of the guided wave, for each frequency, the responses obtained by the transducers can be sparsely represented based on an overcomplete dictionary matrix constructed using multiple discretized wavenumbers and known distances. A CSBL algorithm was developed to infer the posterior probability density function of the weight vector in the sparse representation. The non-zero elements in the sparse weight vector mean that the corresponding dictionary atoms indicated wavenumbers are contained in the frequency response, and the velocity–frequency curve can be finally derived from the wavenumber–frequency curve. The proposed CSBL method has a satisfactory capability to solve this sparse representation of the complex frequency response because a hierarchical form of the Laplace prior is employed to achieve a high degree of sparsity of the weight vector. This method effectively incorporates the real and imaginary parts of the complex frequency response by employing the same hyperparameter to integrate the known information. This method requires only a few randomly placed transducers; thus, it has a wide range of applications. The effectiveness of the proposed approach was validated using multiple guided-wave signals obtained through numerical simulations and an experimental study on a plate structure.

A critical assessment of sparse PCA (research): why (one should acknowledge that) weights are not loadings

Principal component analysis (PCA) is an important tool for analyzing large collections of variables. It functions both as a pre-processing tool to summarize many variables into components and as a method to reveal structure in data. Different coefficients play a central role in these two uses. One focuses on the weights when the goal is summarization, while one inspects the loadings if the goal is to reveal structure. It is well known that the solutions to the two approaches can be found by singular value decomposition; weights, loadings, and right singular vectors are mathematically equivalent. What is often overlooked, is that they are no longer equivalent in the setting of sparse PCA methods which induce zeros either in the weights or the loadings. The lack of awareness for this difference has led to questionable research practices in sparse PCA. First, in simulation studies data is generated mostly based only on structures with sparse singular vectors or sparse loadings, neglecting the structure with sparse weights. Second, reported results represent local optima as the iterative routines are often initiated with the right singular vectors. In this paper we critically re-assess sparse PCA methods by also including data generating schemes characterized by sparse weights and different initialization strategies. The results show that relying on commonly used data generating models can lead to over-optimistic conclusions. They also highlight the impact of choice between sparse weights versus sparse loadings methods and the initialization strategies. The practical consequences of this choice are illustrated with empirical datasets.

Filter pruning via feature map clustering

With the help of network compression algorithms, deep neural networks can be applied on low-power embedded systems and mobile devices such as drones, satellites, and smartphones. Filter pruning is a sub-direction of network compression research, which reduces memory and computational consumption by reducing the number of parameters of model filters. Previous works utilized the “more-simple-less-important” criterion for pruning filters. That is, filters with the smaller norm or more sparse weights in the network are preferentially pruned. In this paper, we found that feature maps are not fully positively correlated with the sparsity of filter weights by observing the visualization of feature maps and the corresponding filters. Hence, we came up with the idea that the priority of filter pruning should be determined by redundancy rather than sparsity. The redundancy of a filter is the measure of whether the output of the filter is repeated with other filters. Based on this, we defined a criterion called redundancy index to rank the filters and introduced it into our filter pruning strategy. Extensive experiments demonstrate the effectiveness of our approach on different model architectures, including VGGNet, GoogleNet, DenseNet, and ResNet. The models compressed with our strategy surpass the state-of-the-art in terms of Floating Point Operations Per Second (FLOPs), parameters reduction, and classification accuracy.

Simple multiple kernel k-means with kernel weight regularization

Multiple kernel clustering (MKC) aims to determine the optimal kernel from several pre-computed basic kernels. Most of MKC algorithms follow a common assumption that the optimal kernel is linearly combined by basic kernels. Based on a min–max framework, a newly proposed MKC method termed simple multiple kernel k-means can acquire a high-quality unified kernel. Although it has achieved promising clustering performance, we have observed that it cannot benefit from any regularization or prior knowledge, thus the learned kernel weight coefficients may be seriously sparse or over-selected. To tackle this issue, we have developed a new algorithm termed simple multiple kernel k-means with kernel weight regularization (SMKKM-KWR), where we introduce average coefficients to avoid too sparse kernel weights. Specially, we add the average kernel coefficients as a regularization term to prevent the learned weight coefficients being far from the average values. After that, an efficient optimization strategy is proposed to solve the new resultant problem. By this way, the unified partition can learn clustering structure from fusion information of all the kernel views, with the goal to reach better clustering performance. Extensive experiments on nine benchmark datasets and four large-scale datasets have demonstrated the effectiveness and efficiency of the proposed algorithm.

Interpretable sparse learned weights and their entropy based quantification for online machine health monitoring

Incipient fault detection and diagnosis provide a firm grounding for machine health monitoring. Nevertheless, fault signatures at the time of an incipient fault are extremely weak and easily submerged by interference. Currently, signal processing based approaches and machine learning based approaches have been studied for incipient fault detection and diagnosis. However, the former requires relevant technicians with expert knowledge, while the latter needs a substantial volume of labeled samples without model interpretability. In this study, a physics-informed learning framework that integrates weight-based sparse degradation modeling with entropy based indicators is proposed to realize online incipient fault detection and diagnosis. Firstly, based on available normal baseline data and real-time data, a weight-based sparse degradation model is proposed to continually update physics-informed model weights so that weak fault characteristics indicated by learned weights can be considerably enhanced. Meanwhile, this study introduces a family of entropy based indicators for machine health monitoring and their performances are thoroughly investigated based on simulation and experimental studies, which aims to quantify amplified fault characteristics revealed by the continuously updated model weights for online incipient fault detection. Two case studies show that the proposed methodology has better detection ability and sensitivity than classical health indicators for incipient bearing faults. Since the proposed methodology does not demand fault data for model establishment, it is closer to real engineering applications and has more engineering meanings. Moreover, physics-informed model weights can automatically capture informative frequencies for immediate diagnosis and further analysis.

ML-Accelerated Yield Analysis Framework Using Regularization for Sparsity in High-Sigma and High-Dimensional Scenarios

Highly repetitive structures in IC, such as SRAM cells typically require extremely low failure ratio, making traditional Monte Carlo analysis extremely time consuming. Furthermore, the “curse of dimensionality” has become a major challenge for existing high-sigma yield analysis techniques. Thus, we propose a “sampling-training-substitution-verification” (STSV) yield analysis framework, which utilizes machine learning (ML) techniques to accelerate yield analysis in high-sigma and high-dimensional scenarios, effectively addresses the “curse of dimensionality.” In our framework, least absolute shrinkage and selection operator (Lasso) regression is adopted to substitute the mapping from process parameters to circuit performance, achieving high accuracy, and generalization. The model is adaptive for both low- and high-dimensional scenarios since the dimensional sparsity is achieved by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$l1$ </tex-math></inline-formula> regularization. In addition, important process parameters can be identified by sparse feature weights of the Lasso model, which is of assistance for yield optimization. Compared with existing yield analysis techniques, the Lasso-based STSV framework offers great saving in a simulation program with integrated circuit emphasis (SPICE) cost, is attractive in high-dimensional demands.

Multiview synthetic aperture radar target recognition method using joint sparse representation and random weight matrix

Multiview synthetic aperture radar (SAR) images could provide richer information for target recognition as reported in previous works. This study exploits multiview SAR images with application to target recognition via the combination of joint sparse representation (JSR) and random weight matrix. First, the multiview SAR images are clustered into several view sets via a coarse clustering algorithm. In this way, the views in each set tend to share high correlations so they can be properly represented with JSR. For the reconstruction errors from different views, they are fused through a random weight matrix, which includes a rich set of linear weight vectors. Afterwards, the fused errors at different choices of weight vectors are analyzed to form a decision value for target recognition, which could better reflect the differences between different classes through multiview SAR images in comparison with traditional JSR-based methods. Therefore, the overall SAR target recognition performance can be effectively enhanced. Experiments are investigated on the moving and stationary target acquisition and recognition dataset under several operating conditions. The results reveal the high effectiveness of the proposed method under standard operating condition and good robustness under typical extended operating conditions including depression angle variance, configuration variants, noise corruption, partial occlusion, and resolution variance.

An omics-to-omics joint knowledge association subtensor model for radiogenomics cross-modal modules from genomics and ultrasonic images of breast cancers

The radiogenomics analysis can provide the connections between genomics and radiomics, which can infer the genomic features of tumors from their radiogenomic associations through the low-cost and non-invasiveness screening ultrasonic images. Although there are a number of pioneer approaches exploring the connections between genomic aberrations and ultrasonic features, these studies mainly focus on the relationship between ultrasonic features and only the most popular cancer genes, confronting two difficulties: missing many-to-many relationships as omics-to-omics view, and confounding group-specific associations with whole sample associations. To overcome the difficulty of omics-to-omics view and the issue of tumor heterogeneity, we propose an omics-to-omics joint knowledge association subtensor model. Specifically, the subtensor factorization framework can successfully discover the joint cross-modal module via an omics-to-omics view, while the sparse weight sample indication strategy can mine sample subgroups from the multi-omic data with tumor heterogeneity. The experimental evaluation result shows the jointness of the discovered modules across omics, their association with tumorigenesis contribution, and their relation for cancer related functions. In summary, our proposed omics-to-omics joint knowledge association subtensor model can serve as an efficient tool for radiogenomic knowledge associations, promoting the cross-modal knowledge graph construction of in explainable artificial intelligence cancer diagnosis.

Tight Compression: Compressing CNN Through Fine-Grained Pruning and Weight Permutation for Efficient Implementation

The unstructured sparsity after pruning poses a challenge to the efficient implementation of deep learning models in existing regular architectures like systolic arrays. On the other hand, coarse-grained structured pruning is suitable for implementation in regular architectures but tends to have higher accuracy loss than unstructured pruning when the pruned models are of the same size. In this work, we propose a model compression method based on a novel weight permutation scheme to fully exploit the fine-grained weight sparsity in the hardware design. Through permutation, the optimal arrangement of the weight matrix is obtained, and the sparse weight matrix is further compressed to a small and dense format to make full use of the hardware resources. Two pruning granularities are explored. In addition to the unstructured weight pruning, we also propose a more fine-grained subword-level pruning to further improve the compression performance. Compared to the state-of-the-art works, the matrix compression rate is significantly improved from <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$5.88\times $ </tex-math></inline-formula> to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$14.13\times $ </tex-math></inline-formula> . As a result, the throughput and energy efficiency are improved by 2.75 and 1.86 times, respectively.

Stratified incomplete local simplex tests for curvature of nonparametric multiple regression

Principled nonparametric tests for regression curvature in $\mathbb{R}^{d}$ are often statistically and computationally challenging. This paper introduces the stratified incomplete local simplex (SILS) tests for joint concavity of nonparametric multiple regression. The SILS tests with suitable bootstrap calibration are shown to achieve simultaneous guarantees on dimension-free computational complexity, polynomial decay of the uniform error-in-size, and power consistency for general (global and local) alternatives. To establish these results, a general theory for incomplete $U$-processes with stratified random sparse weights is developed. Novel technical ingredients include maximal inequalities for the supremum of multiple incomplete $U$-processes.

Sparse oblique decision trees: a tool to understand and manipulate neural net features

The widespread deployment of deep nets in practical applications has lead to a growing desire to understand how and why such black-box methods perform prediction. Much work has focused on understanding what part of the input pattern (an image, say) is responsible for a particular class being predicted, and how the input may be manipulated to predict a different class. We focus instead on understanding which of the internal features computed by the neural net are responsible for a particular class. We achieve this by mimicking part of the neural net with an oblique decision tree having sparse weight vectors at the decision nodes. Using the recently proposed Tree Alternating Optimization (TAO) algorithm, we are able to learn trees that are both highly accurate and interpretable. Such trees can faithfully mimic the part of the neural net they replaced, and hence they can provide insights into the deep net black box. Further, we show we can easily manipulate the neural net features in order to make the net predict, or not predict, a given class, thus showing that it is possible to carry out adversarial attacks at the level of the features. These insights and manipulations apply globally to the entire training and test set, not just at a local (single-instance) level. We demonstrate this robustly in the MNIST and ImageNet datasets with LeNet5 and VGG networks.

ORSAS: An Output Row-Stationary Accelerator for Sparse Neural Networks

Various pruning skills and network compression methods make modern neural networks sparse for both weights and activations. However, GPUs (graphics processing units) and most of the customized CNN (convolutional neural networks) accelerators have not taken advantage of the sparsity of neural networks, and the accelerators for sparse neural networks in these years are suffering from low computational resource utilization. This paper first proposes an output row-stationary dataflow that exploits the sparsity of both weights and activations. It allows the accelerator to process weights and activations in their compressed form, leading to the high utilization of computational resources, i.e. multipliers. Besides, a low-cost compression algorithm is adopted for both weights and input activations to reduce the power consumption of data access. Secondly, the Y-buffer is proposed to eliminate repeated reading of input activations caused by halo effects, which emerge when tiling large-size input feature maps (ifmaps). Third, an interleaved broadcasting mechanism is introduced to alleviate the imbalance problems caused by the irregularity of sparse data. Finally, a prototype design called ORSAS is synthesized and placed and routed with a SMIC 55 nm process. The evaluation results show that ORSAS occupies the smallest logical cell area, and has the highest multiplier utilization among the peers’ works when the sparsity is large. ORSAS keeps the utilization of multipliers in a range from 60% to 90% over the convolutional layers of popular sparse CNNs and achieves ultra-low power consumption and highest efficiency.

Generalization bounds for sparse random feature expansions

Random feature methods have been successful in various machine learning tasks, are easy to compute, and come with theoretical accuracy bounds. They serve as an alternative approach to standard neural networks since they can represent similar function spaces without a costly training phase. However, for accuracy, random feature methods require more measurements than trainable parameters, limiting their use for data-scarce applications or problems in scientific machine learning. This paper introduces the sparse random feature expansion to obtain parsimonious random feature models. Specifically, we leverage ideas from compressive sensing to generate random feature expansions with theoretical guarantees even in the data-scarce setting. In particular, we provide generalization bounds for functions in a certain class (that is dense in a reproducing kernel Hilbert space) depending on the number of samples and the distribution of features. The generalization bounds improve with additional structural conditions, such as coordinate sparsity, compact clusters of the spectrum, or rapid spectral decay. In particular, by introducing sparse features, i.e. features with random sparse weights, we provide improved bounds for low order functions. We show that the sparse random feature expansions outperforms shallow networks in several scientific machine learning tasks.

Es-Fedavg: Early-Exit Personalized Federated Learning with Sparse Weight for Adaptive Computation

Es-Fedavg: Early-Exit Personalized Federated Learning with Sparse Weight for Adaptive Computation

PTTS: Power-aware tensor cores using two-sided sparsity

Deep Neural networks (DNNs) have become the compelling solution for a broad range of applications such as automatic translation, advertisement recommendation, and speech recognition. Matrix multiplication is the fundamental operation used in various classes of DNNs. Introduction of tensor cores (TCs) in NVIDIA GPGPUs particularly aims at acceleration of neural networks. A TC is a specialized unit dedicated to compute matrix-multiply-and-accumulate (MMA) operations. State-of-the-art DNNs are known to be power-hungry and compute-intensive due to increasing depth of networks, i.e. multiple layers with massive number of neurons. This causes excessive energy in TCs.Sparse neural networks have emerged as an effective solution to address massive amount of computations in DNNs. While sparsity is widely used for acceleration of DNNs, only a handful of studies focused on energy aspect of sparse DNNs and energy-efficient architectures for TCs are scarce. We exploit sparsity in DNNs and propose power gating multipliers with sparse activations or weights. We show that sparsity occurs in TCs for short intervals and conventional power gating techniques cannot fully exploit these idle intervals mainly due to overhead of power gating. To mitigate the overhead of power gating, we propose power-aware tensor cores using two-sided sparsity (PTTS) which monitors inputs of multipliers and turns them off only if inputs remain sparse for long intervals. We also propose an architectural technique that shuffles multiplications to pack idle intervals and increase opportunities for power gating. We introduce a low-cost and implementation-efficient sparse input operand interconnect to change order of effectual multiplications. Our proposed techniques combined are able to save energy by 68% in Tensor Cores with negligible impact on performance while maintaining accuracy.

ESSA: Design of a Programmable Efficient Sparse Spiking Neural Network Accelerator

Spiking neural networks (SNNs) have been witnessing the developing trends to reduce the model size and improve the hardware efficiency for area- and energy-based applications, which are processed by model pruning and data compressions. However, it is challenging to exploit the unstructured sparsity of SNNs for the dense neuromorphic processors. In this article, we present an efficient sparse SNN accelerator (ESSA), which leverages both the temporal sparsity of spike events and the spatial sparsity of weights in SNN inference. It provides both the compressed weights for sparse SNNs and the uncompressed weights for compact SNNs. The self-adaptive spike compression is proposed for sparse spike scenarios, leading to the improvement of throughput by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$3.2\times $ </tex-math></inline-formula> . ESSA executes a flexible fan-in–fan-out tradeoff by using combinable dendrites, which overcomes the fan-in limitation in neuromorphic systems. Furthermore, a low-latency intrachip spike multicast method is adopted to reduce the resource overhead. Implemented on the Xilinx Kintex Ultrascale field-programmable gate array (FPGA), ESSA achieves an equivalent performance of 253.1 GSOP/s and an energy efficiency of 32.1 GSOP/W for 75% weight sparsity at 140 MHz. The implementation of a four-layer fully connected SNN is expected to perform <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2.6~\mu \text{s}$ </tex-math></inline-formula> per time step and the energy consumption is <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$14.6~\mu \text{J}$ </tex-math></inline-formula> . Our results demonstrate that ESSA outperforms several state-of-the-art application-specific integrated circuit (ASIC) or FPGA neuromorphic processors.

An Efficient Unstructured Sparse Convolutional Neural Network Accelerator for Wearable ECG Classification Device

Convolution neural network (CNN) with pruning techniques has shown remarkable prospects in electrocardiogram (ECG) classification. However, efficiently deploying the existing pruned neural network to wearable devices for ECG classification is a great challenge due to the limited hardware resource and randomly distributed sparse weights. To address this issue, an efficient unstructured sparse CNN accelerator is proposed in this paper. A tile-first dataflow with compressed data storage format is presented to skip zero weight multiplications and increase the computing efficiency during inference of small-scale model with large sparsity. The two-level weight index matching structure in the dataflow exploits shifting operation to select valid data pairs and maintain the fully-pipelined calculation process. A configurable processing element (PE) array with 32-bit instruction control is proposed to increase the flexibility of the accelerator. Verified in FPGA and post-synthesis simulations in SMIC 40nm process, the proposed sparse CNN accelerator consumes <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$3.93~\mu $ </tex-math></inline-formula> J/classification at 2MHz clock frequency and it achieves an averaged ECG classification accuracy of 98.99%. A computing efficiency of 118.75% is realized which is improved by 48% compared to the dense baseline. In brief, the proposed efficient CNN accelerator is especially suitable for wearable ECG classification device.