Zero Weight Research Articles

Adaptively Sparse Transformers Hawkes Process

Nowadays, many sequences of events are generated in areas as diverse as healthcare, finance, and social network. People have been studying these data for a long time. They hope to predict the type and occurrence time of the next event by using relationships among events in the data. recently, with the successful application of Recurrent Neural Network (RNN) in natural language processing, it has been introduced into point process. However, RNN cannot capture the long-term dependence among events well, and self-attention can partially mitigate this problem precisely. Transformer Hawkes Process (THP) using self-attention greatly improves the performance of the Hawkes Process, but THP cannot ignore the effect of irrelevant events, which will affect the computational complexity and prediction accuracy of the model. In this paper, we propose an Adaptively Sparse Transformers Hawkes Process (ASTHP). ASTHP considers the periodicity and nonlinearity of event time in the time encoding process. The sparsity of the ASTHP is achieved by substituting Softmax with [Formula: see text]-entmax: [Formula: see text]-entmax is a differentiable generalization of Softmax that allows unrelated events to gain exact zero weight. By optimizing the neural network parameters, different attention heads can adaptively select sparse modes (from Softmax to Sparsemax). Compared with the existing models, ASTHP model not only ensures the prediction performance but also improves the interpretability of the model. For example, the accuracy of ASTHP model on MIMIC-II dataset is improved by nearly 3 percentage points, and the model fitting degree and stability are also improved significantly.

Employment reconciliation and nowcasting

SummaryWe construct a latent employment estimate for the United States, which both reconciles the information from separate payroll and household surveys and incorporates the preliminary data revision process of the payroll data. We find that our reconciled latent employment series looks somewhat different than the initial release of payroll employment and is closer to the fully revised data that is benchmarked to a near census of employment. A real‐time exercise, however, suggests that the reconciled employment estimate is remarkably similar to the initial release of payroll employment with near zero weight on the household survey information.

FAT: An In-Memory Accelerator With Fast Addition for Ternary Weight Neural Networks

Convolutional neural networks (CNNs) demonstrate excellent performance in various applications but have high computational complexity. Quantization is applied to reduce the latency and storage cost of CNNs. Among the quantization methods, binary and ternary weight networks (BWNs and TWNs) have a unique advantage over 8 and 4-bit quantization. They replace the multiplication operations in CNNs with additions, which are favored on in-memory-computing (IMC) devices. IMC acceleration for BWNs has been widely studied. However, though TWNs have higher accuracy and better sparsity than BWNs, IMC acceleration for TWNs has limited research. TWNs on the existing IMC devices are inefficient because the sparsity is not well utilized, and the addition operation is not efficient. In this article, we propose FAT as a novel IMC accelerator for TWNs. First, we propose a sparse addition control unit, which utilizes the sparsity of TWNs to skip the null operations on zero weights. Second, we propose a fast addition scheme based on the memory sense amplifier (SA) to avoid the time overhead of both carry propagation and writing back the carry to memory cells. Third, we further propose a combined-stationary data mapping to reduce the data movement of activations and weights and increase the parallelism across memory columns. Simulation results show that for addition operations at the SA level, FAT achieves <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2.00\times $ </tex-math></inline-formula> speedup, <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.22\times $ </tex-math></inline-formula> power efficiency, and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.22\times $ </tex-math></inline-formula> area efficiency compared with a state-of-the-art IMC accelerator ParaPIM. FAT achieves <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$10.02\times $ </tex-math></inline-formula> speedup and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$12.19\times $ </tex-math></inline-formula> energy efficiency compared with ParaPIM on networks with 80% average sparsity.

Absolute and relative weight restrictions in DEA - a comparison

In most data envelopment analysis (DEA) models, the decision maker (DM) only selects inputs and outputs and does not have to make any previous assumptions about the underlying production process. Without additional weight restrictions, the linear program may assign zero weights to inputs and outputs of certain decision-making units (DMUs). Specialised DMUs can be calculated as efficient because only subsets of the data are included in their efficiency calculation. In most cases, however, it is not desirable to ignore inputs or outputs previously selected by the DM, so technical efficiency can overestimate true efficiency. Additional weight restrictions can avoid specialised DMUs, further improve discrimination between DMUs, and allow the implementation of prior knowledge of the production process. This publication provides a comprehensive literature review of relevant studies on additional weight restrictions in the DEA and discusses their implementation and effects. The latter aspects are demonstrated first for five artificial DMUs and then for 1,000,000 simulated DMUs.

An Efficient Unstructured Sparse Convolutional Neural Network Accelerator for Wearable ECG Classification Device

Convolution neural network (CNN) with pruning techniques has shown remarkable prospects in electrocardiogram (ECG) classification. However, efficiently deploying the existing pruned neural network to wearable devices for ECG classification is a great challenge due to the limited hardware resource and randomly distributed sparse weights. To address this issue, an efficient unstructured sparse CNN accelerator is proposed in this paper. A tile-first dataflow with compressed data storage format is presented to skip zero weight multiplications and increase the computing efficiency during inference of small-scale model with large sparsity. The two-level weight index matching structure in the dataflow exploits shifting operation to select valid data pairs and maintain the fully-pipelined calculation process. A configurable processing element (PE) array with 32-bit instruction control is proposed to increase the flexibility of the accelerator. Verified in FPGA and post-synthesis simulations in SMIC 40nm process, the proposed sparse CNN accelerator consumes <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$3.93~\mu $ </tex-math></inline-formula> J/classification at 2MHz clock frequency and it achieves an averaged ECG classification accuracy of 98.99%. A computing efficiency of 118.75% is realized which is improved by 48% compared to the dense baseline. In brief, the proposed efficient CNN accelerator is especially suitable for wearable ECG classification device.

Multi-label feature selection via robust flexible sparse regularization

Multi-label feature selection is an efficient technique to deal with the high dimensional multi-label data by selecting the optimal feature subset. Existing researches have demonstrated that l1-norm and l2,1-norm are promising roles for multi-label feature selection. However, two important issues are ignored when existing l1-norm and l2,1-norm based methods select discriminative features for multi-label data. First, l1-norm can enforce sparsity on each feature across all instances while numerous selected features lack discrimination due to the generated zero weight values. Second, l2,1-norm not only neglects label-specific features but also ignores the redundancy among features. To this end, we design a Robust Flexible Sparse Regularization norm (RFSR), furthermore, proposing a global optimization framework named Robust Flexible Sparse regularized multi-label Feature Selection (RFSFS) based on RFSR. Finally, an efficient alternating multipliers based optimization scheme is developed to iteratively optimize RFSFS. Empirical studies on fifteen benchmark multi-label data sets demonstrate the effectiveness and efficiency of RFSFS.

Search-Free Inference Acceleration for Sparse Convolutional Neural Networks

Sparse convolution neural networks (CNNs) are promising in reducing both memory usage and computational complexity while still preserving high inference accuracy. State-of-the-art sparse CNN accelerators can deliver high throughput by skipping zero weights and/or activations. To operate on only nonzero weights and activations, sparse accelerators typically search pairs of nonzero weights and activations for multiplication-accumulation (MAC) operations. However, the conventional search operation results in a severe limitation in the processing element (PE) array scale because of the enormous demands of internal interconnection and memory bandwidth. In this article, we first provide a design principle to free the search process of sparse CNN accelerations. Specifically, the indexes of the static compressed weights access the dynamic activations directly to avoid the search process for MAC operations. We then develop two search-free inference accelerators, called Swan and Swan-flexible, for sparse CNN accelerations. Swan supports search-free sparse convolution accelerations for interconnection and bandwidth saving. Compared with Swan, Swan-flexible not only has the search-free capability but also comprises a configurable architecture for optimum throughput. We formulate a mathematical optimization problem by combining the configurable characterization with the compressive dataflow to optimize the overall throughput. Evaluations based on a place-and-route process show that the proposed designs, in a compact factor of 4096 PEs, achieve 1.5– <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2.7\times $ </tex-math></inline-formula> higher speedup and 6.0– <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$13.6\times $ </tex-math></inline-formula> better energy efficiency than representative accelerator baselines with the same PE array scale.

Weyl Group Characters Afforded By Zero Weight Spaces

Let G be a compact Lie group with Weyl group W. We give a formula for the character of W on the zero weight space of any finite-dimensional representation of G. The formula involves weighted partition functions, generalizing Kostant’s partition function. On the elliptic set of W, the partition functions are trivial. On the elliptic regular set, the character formula is a monomial product of certain coroots, up to a constant equal to 0 or ± 1. This generalizes Kostant’s formula for the trace of a Coxeter element on a zero weight space. If the long element w0 = − 1, our formula gives a method for determining all representations of G for which the zero weight space is irreducible.

DNN pruning with principal component analysis and connection importance estimation

DNN pruning reduces memory footprint and computational work of DNN-based solutions to improve performance and energy-efficiency. An effective pruning scheme should be able to systematically remove connections and/or neurons that are unnecessary or redundant, reducing the DNN size without any loss in accuracy. In this paper we show that some of the most popular pruning schemes, such as the Near Zero Weights, require an extremely time-consuming iterative process that requires retraining the DNN many times to tune the pruning hyperparameters. Then, we propose a DNN pruning scheme based on Principal Component Analysis and relative importance of each neuron’s connection (PCA+DIRIE) that automatically finds the optimized DNN in one shot without requiring hand-tuning of multiple parameters. The experimental results show the effectiveness of our method on several benchmarks. Notably, on ImageNet, PCA+DIRIE can prune up to 60% of ResNet-50 with negligible impact on accuracy.

Adjusted maximum likelihood method for multivariate Fay–Herriot model

For the Fay–Herriot model, the empirical best linear unbiased predictor (EBLUP) is the weighted sum of direct estimate and regression-synthetic estimate, where the weight depends on the estimate of variance component of random area effects. However, in some cases, the weight of direct survey estimator is zero and the EBLUP reduces to regression-synthetic estimator. This leads to unfavourable estimates. Therefore, Lahiri and Li (2009) introduced the concept of adjusted maximum likelihood method for linear mixed models. The method was then generalized in Li and Lahiri (2010) and Yoshimori and Lahiri (2014) to develop adjusted maximum likelihood consistent variance estimators with positive variance estimates. These adjustments prevent zero weight of direct estimate. In this paper, we extend their methods to propose adjusted maximum likelihood estimates for multivariate Fay–Herriot models. Moreover, we derive their asymptotic properties and perform numerical results to investigate performances of the proposed estimates.

The sooner the better: lives saved by the lockdown during the COVID-19 outbreak. The case of Italy

SummaryThis paper estimates the effects of non-pharmaceutical interventions – mainly, the lockdown – on the COVID-19 mortality rate for the case of Italy, the first Western country to impose a national shelter-in-place order. We use a new estimator, the augmented synthetic control method (ASCM), that overcomes some limits of the standard synthetic control method (SCM). The results are twofold. From a methodological point of view, the ASCM outperforms the SCM in that the latter cannot select a valid donor set, assigning all the weights to only one country (Spain) while placing zero weights to all the remaining. From an empirical point of view, we find strong evidence of the effectiveness of non-pharmaceutical interventions in avoiding losses of human lives in Italy: conservative estimates indicate that the policy saved in total more than 21,000 human lives.

Classifier Performance Evaluation for Lightweight IDS Using Fog Computing in IoT Security

In this article, a Host-Based Intrusion Detection System (HIDS) using a Modified Vector Space Representation (MVSR) N-gram and Multilayer Perceptron (MLP) model for securing the Internet of Things (IoT), based on lightweight techniques and using Fog Computing devices, is proposed. The Australian Defence Force Academy Linux Dataset (ADFA-LD), which contains exploits and attacks on various applications, is employed for the analysis. The proposed method is divided into the feature extraction stage, the feature selection stage, and classification modeling. To maintain the lightweight criteria, the feature extraction stage considers a combination of 1-gram and 2-gram for the system call encoding. In addition, a Sparse Matrix is used to reduce the space by keeping only the weight of the features that appear in the trace, thus ignoring the zero weights. Subsequently, Linear Correlation Coefficient (LCC) is utilized to compensate for any missing N-gram in the test data. In the feature selection stage, the Mutual Information (MI) method and Principle Component Analysis (PCA) are utilized and then compared to reduce the number of input features. Following the feature selection stage, the modeling and performance evaluation of various Machine Learning classifiers are conducted using a Raspberry Pi IoT device. Further analysis of the effect of MLP parameters, such as the number of nodes, number of features, activation, solver, and regularization parameters, is also conducted. From the simulation, it can be seen that different parameters affect the accuracy and lightweight evaluation. By using a single hidden layer and four nodes, the proposed method with MI can achieve 96% accuracy, 97% recall, 96% F1-Measure, 5% False Positive Rate (FPR), highest curve of Receiver Operating Characteristic (ROC), and 96% Area Under the Curve (AUC). It also achieved low CPU time usage of 4.404 (ms) milliseconds and low energy consumption of 8.809 (mj) millijoules.

Weight zero in tensor-decomposable irreducible representations of simple algebraic groups

Let G be a simple algebraic group in defining characteristic p > 0 , and let V be an irreducible G -module which is the tensor product of exactly two non-trivial modules. We obtain a criterion for V to have the zero weight. In addition, we provide a uniform criterion for an irreducible representation of a simple Lie algebra over the complex numbers to have a multiple of a prescribed fundamental weight.

A Practical Highly Paralleled ReRAM-Based DNN Accelerator by Reusing Weight Pattern Repetitions

Resistive random access memory (ReRAM)-based processing-in-memory (PIM) architecture has been designed to accelerate deep neural networks (DNNs) by concurring computation and memory barriers. To further improve memory and computation efficiency, the weight sparsity characteristic has been explored to optimize the ReRAM-based DNN accelerators. However, these designs only focus on compressing zero weights to eliminate ineffectual computation. In this article, we thoroughly analyze the weight distribution characteristics of several typical DNN models and observe many nonzero weight pattern repetitions (WPRs). Therefore, there is an opportunity to further improve the performance and energy efficiency by reusing these WPR. We propose a novel ReRAM-based accelerator—PattPIM, to achieve space compression and computation reuse by exploring DNN WPR based on practical ReRAM crossbars. In PattPIM, we propose a configurable WPR-aware DNN engine and a WPR-to-OU mapping scheme to save both space and computation resources. An intraprocessing engine (PE) pipeline is designed to improve the parallelism of the computation process. Furthermore, we adopt an approximate weight pattern transform algorithm to improve the DNN WPR ratio to enhance the reuse efficiency with negligible accuracy loss. Our evaluation with 6 DNN models shows that the proposed PattPIM delivers significant performance improvement, ReRAM resource efficiency and energy saving.

Exploring the reasons for efficiency in Spanish airports

In recent years the Spanish government has invested significantly in the infrastructure of airports. It is not clear if this investment has been efficiently applied. Airports operate as independent profit centres but are under the control of a central authority, AENA. We study Spanish airport efficiency using Data Envelopment Analysis (DEA). In standard studies, DEA summarises the efficiency of a unit by means of a single number. Here we go beyond the efficiency score by combining DEA with multivariate analysis techniques. In this way we are able to establish why a particular airport reaches a given efficiency level, and what is its approach to the use of resources and the achievement of results. The combined use of DEA and multivariate statistical analysis overcomes the well-known zero weights problem and permits the visualisation of the results and the addition of qualitative information to the interpretation of the results. On this basis, for each unit of assessment (airport), individual managerial recommendations are drawn. Additionally, the visualisation allows differentiating between units more efficient in certain operations compared to others (specialisation).

Differential co-expression analysis reveals early stage transcriptomic decoupling in alzheimer\u2019s disease

BackgroundAlzheimer’s disease (AD) is one of the leading causes of death in the US and there is no validated drugs to stop, slow or prevent AD. Despite tremendous effort on biomarker discovery, existing findings are mostly individual biomarkers and provide limited insights into the transcriptomic decoupling underlying AD. We propose to explore the gene co-expression patterns in multiple AD stages, including cognitively normal (CN), early mild cognitive impairment (EMCI), late MCI and AD.MethodsWe modified traiditonal joint graphical lasso to model our asusmption that the co-expression networks in consecutive disease stages are largely similar with critical differences. In addition, we performed subsequent network comparison analysis for identification of stage specific transcriptomic decoupling. We focused our analysis on top AD-enriched pathways.ResultsWe observed that 419 edges in CN, 420 edges in EMCI, 381 edges in LMCI and 250 edges in AD were frequently estimated with non zero weights. With modified JGL, the weight of all estimated edges in CN, EMCI and LMCI are zero. In AD group, 299 edges were occasionally estimated to be nonzero and the average correlation between genes was 0.0023. For co-expression change during AD progression, there are 66 pairs of genes that demonstrated a continuously decreasing or increasing co-expression from CN to EMCI, LMCI and AD.The network level clustering coefficient remains stable from CN to LMCI and then decreases significantly when progressing to AD. When evaluating edge level differences, we identified eight gene modules with continuously decreasing or increasing co-expression patterns during AD progression. Five of them shows significant changes from CN to EMCI and thus have the potential to serve system biomarkers for early screening of AD.ConclusionWe employed a modified joint graphical lasso for estimation of co-expression networks for multiple stages of AD. Comparing with graphical lasso, our modified joint graphical lasso model accounts for the similarity in consecutive disease stages. Our results on real data set revealed five gene clusters with obvious co-expression pattern change from CN to EMCI, which could be used as potential system-level biomarkers for early screening of AD.

Attitude Estimation Based on Robust Information Cubature Quaternion Filter

We propose a robust attitude estimation method for multisensor systems with heavy-tailed noise. The attitude model is represented by a quaternion, and the propagation process in the filter is based on a gyro-based model. To avoid the constraint of quaternion, the quaternion is converted to modified Rodrigues parameters during the prediction and update process. Instead of the minimum mean square error principle, the maximum correntropy criterion is applied in the update process, which is robust to heavy-tailed noises. Considering the linear form in the update process, the information filtering algorithm is applied in the proposed method to simplify the calculation of multisensor information fusion. Moreover, the application of information filtering algorithm can also avoid the numerical problem that zero weight functions cannot be incorporated into the Gaussian filtering framework. Simulations for several cases are performed, and the simulation results indicate the robustness and efficiency of the proposed filter.

Avoiding dissimilarity between the weights of the optimal DEA solutions

The zero weights in data envelopment analysis evaluation causes some problems such as ignoring the some inputs and/or outputs of DMUs under evaluation. Moreover, some authors claimed that the great differences in weights might be a problem. The aim of this paper is to extend the multiplier bound approach to avoid zero weights and great differences in the values of multipliers more. We show that our proposed model is equivalent to the type I assurance region model that will be used in the evaluation efficiency.

Performing Sparse Regularization and Dimension Reduction Simultaneously in Multimodal Data Fusion.

Collecting multiple modalities of neuroimaging data on the same subject is increasingly becoming the norm in clinical practice and research. Fusing multiple modalities to find related patterns is a challenge in neuroimaging analysis. Canonical correlation analysis (CCA) is commonly used as a symmetric data fusion technique to find related patterns among multiple modalities. In CCA-based data fusion, principal component analysis (PCA) is frequently applied as a preprocessing step to reduce data dimension followed by CCA on dimension-reduced data. PCA, however, does not differentiate between informative voxels from non-informative voxels in the dimension reduction step. Sparse PCA (sPCA) extends traditional PCA by adding sparse regularization that assigns zero weights to non-informative voxels. In this study, sPCA is incorporated into CCA-based fusion analysis and applied on neuroimaging data. A cross-validation method is developed and validated to optimize the parameters in sPCA. Different simulations are carried out to evaluate the improvement by introducing sparsity constraint to PCA. Four fusion methods including sPCA+CCA, PCA+CCA, parallel ICA and sparse CCA were applied on structural and functional magnetic resonance imaging data of mild cognitive impairment subjects and normal controls. Our results indicate that sPCA significantly can reduce the impact of non-informative voxels and lead to improved statistical power in uncovering disease-related patterns by a fusion analysis.

Robust information unscented particle filter based on M‐estimate

Herein, inspired by the unscented particle filter (UPF), an improved particle filter (PF) using an unscented information filter based on M-estimate (UIF-BM) to approximate the importance density function (IDF) is designed. The filtering method is proposed to obtain an accurate state estimation for a non-linear discrete time dynamical system with non-Gaussian system noises and observation noises contaminated by some Gaussian impulsive noises. The PF framework is used as a common method to handle non-Gaussian system noises. In order to obtain a precise IDF, UIF based on M-estimate is used for heavy-tailed observation noises. Meanwhile, instead of the robust Gaussian filter, the robust IF can avoid the numerical problem that zero weight functions cannot be incorporated into the framework. The simulation results indicate the estimation accuracy and efficiency of the proposed filter. Compared with the UIF-BM, PF, and UPF, the superiority of the proposed filter against the non-ideal system and observation noises is obvious.